Skip to main content

Create and deploy a project

After engineers configure TapData connections, tasks, and APIs, they can package those resources as a project, export the configuration, and deploy it to another environment. Deployment can run automatically through GitHub Actions or manually through file export and import.

tip

This guide covers both automated deployment and manual import/export. If you plan to use GitHub and GitHub Actions, first complete Set up an automated deployment pipeline. If you only need manual import/export, follow this guide and see Appendix: Manually import configuration.

Example scenario​

This guide uses a common real-time data warehouse scenario. A team synchronizes data from an Oracle source database to a Doris warehouse. The team has already verified wide-table synchronization tasks and an external API in the development environment. The next step is to promote the same configuration to a testing or acceptance environment, and then to production.

The following workflow shows how to create a project, export configuration, deploy automatically, and publish manually when needed.

tip

Use the same connection names across environments. Use names that contain only letters, numbers, and underscores, such as oracle_source. During automated deployment, TapData matches connection names to GitHub Secrets and Variables, then injects the real address, username, and password for the target environment.

Step 1: Create a project and select resources​

Package the team's tasks and API as one project. The project becomes the unit that you export, review, deploy, and roll back.

  1. Log in to the TapData console. In the left navigation pane, choose Advanced Settings > Project Management.

  2. At the top of the left panel, click + to create a project. Enter a project name, such as dw-pipeline. Use the same name as the GitHub tenant repository when possible.

  3. In the middle panel, switch between Migrate Task, Transformation Task, and API. Select CRM_TO_DW, ORDER_TO_DW, and customer-api, then click Add selected > to move them to the selected list.

    Create a project and select resources

    Dependent connections and Serving Indexes
    • Dependent connections: When you select tasks or APIs, TapData automatically includes dependent connections. In this example, oracle_source and doris_target are included automatically.
    • Managing Serving Indexes: If your project includes data service APIs and you want the target environment to create the related MongoDB Serving Indexes, open the API's Serving Indexes page in the source environment. Load the indexes and select the indexes that TapData should manage. Only selected index declarations are exported with the project and applied during automated deployment. The target database account must have permission to create indexes. If it does not, a DBA must create them manually based on the deployment report.

  4. Click Save.

Step 2: Connect the Git repository​

tip

If you do not want to integrate with GitHub, skip this step. In Step 3, choose file export and import the archive manually. If you use Git export, make sure the repository URL is writable and the personal access token has permission to write contents and create Pull Requests in the tenant repository.

Connect the TapData project to the GitHub tenant repository. After the connection is configured, exports can be pushed to the repository and opened as Pull Requests without downloading and uploading files manually.

  1. In the upper-right corner, click Git Configuration.

  2. In the dialog box, enter the GitHub tenant repository URL and personal access token.

    Configure the Git repository

  3. Click Save.

Step 3: Export the configuration​

Export the project configuration from the development environment and submit it to GitHub for later deployment.

  1. On the Project Management page, click Export in the upper-right corner. In the export dialog box, select the project to export.

  2. Select an Export type.

    Export a project

    • Git Export: Available after a Git repository is connected. TapData pushes the configuration files to GitHub and creates a Pull Request. Enter the following information:

      FieldDescription
      Branch nameThe system generates a branch name that starts with feat_ and includes a timestamp. You can edit the branch name.
      PR titleA short summary of the change for review.
      PR descriptionOptional. Describe why the change is needed and what it affects.

    • File Export: Downloads the configuration as a compressed archive. Use this option when Git integration is not configured. For the import steps, see Appendix: Manually import configuration.

  3. In the resource list, review the tasks and APIs to be exported. If the list is correct, click Confirm Export.

    Credential masking and rerun behavior
    • Credential masking: Git export always masks sensitive credentials, including database passwords and connection strings, before writing files to the repository. During later deployments, GitHub Environment values inject the credentials. File export retains the complete configuration for offline manual migration.
    • Rerun: Enable Rerun only when the target environment needs the task to run a full synchronization again, for example after adding source tables or changing primary keys. For routine changes, keep the default setting so tasks continue from the last checkpoint.
Exported file structure

Exported configuration is organized as a directory. Git export commits this directory to the repository. File export packages it as an archive.

{project-name}_tapdata_export/
├── GroupInfo.json          # Project metadata: project name, Git repository, and resource list
├── Connection/             # Connection configuration, including dependencies of tasks and APIs
│   ├── {id}_Connection_Config.json    # Masked for Git export; file export might contain sensitive information
│   └── {id}_Connection_Metadata.json  # Table metadata for the connection
├── Task/                   # Task configuration
│   ├── {id}_MigrateTask.json          # Data replication task
│   └── {id}_SyncTask.json             # Data transformation task
├── API/                    # API configuration
│   ├── {id}_Module.json               # API definition, including path, fields, and query logic
│   └── MetadataDefinition.json
└── User/                   # User and role information for restoring operator context
    ├── Users.json
    ├── Roles.json
    ├── RoleMappings.json
    └── UserIdEmailMap.json

Notes:

  • Connections: TapData detects and exports connections based on task and API dependencies. You do not need to select them manually.
  • Sensitive information: Git export clears usernames, passwords, and other credential fields. During automated deployment, TapData injects real values from GitHub Secrets and Variables. File export retains the complete configuration, so the exported archive might contain sensitive information. Store it securely. For manual import, update connection values as needed for the target environment.
  • User data: The User directory contains operator account and role information so the target environment can restore the user context. Passwords are stored as hashes and do not include plaintext values.

Step 4: Optional: Merge the Pull Request to deploy to development validation​

If the dev environment is configured, merge the Pull Request in GitHub to deploy the exported configuration to the development validation environment. This step verifies that the configuration files can be imported before you promote them further. If your process only uses testing and production, skip this stage and adjust the tenant deployment workflow accordingly.

  1. In the GitHub tenant repository, open the Pull Request created by TapData. Review the exported configuration, then click Merge.

  2. The merge triggers the GitHub Actions TapData Deploy workflow and deploys the configuration to the development validation environment.

  3. If the preview shows changes to connections, tasks, or APIs, approve the deploy gate on the Actions page.

    Review the deployment preview report

    The workflow writes a structured add, update, and delete preview to the run Summary page. Review the following areas before approving the deployment:

    • Connections: Verify that connection settings match the target environment.
    • Tasks: Review additions, removals, and configuration changes for DAG nodes and edges, including From and To values.
    • APIs: Review API paths and exposed fields. When only Serving Index declarations change, the report marks the API as (serving-index declarations only) to show that the API contract is unchanged.
    • Serving Indexes: Review the planned MongoDB Serving Indexes, including names, fields, and sort directions. If the target database account cannot create indexes, a DBA can run the createIndex commands in the report. Use the import result after deployment as the source of truth. If a related synchronization task uses dropTable, rerunning the task also removes newly created Serving Indexes.
    • Orphan indexes: Review indexes that exist in a target collection but are not declared by any API. They can increase write overhead and count toward MongoDB's limit of 64 indexes per collection. A DBA must verify and remove them manually when appropriate.

  4. After deployment finishes, sign in to the development validation TapData environment. Verify that CRM_TO_DW, ORDER_TO_DW, and customer-api were imported correctly and that the connections pass the connection test.

Step 5: Create a tag to deploy to testing or acceptance​

After the configuration is ready for the next validation environment, create and push a Git tag. The official template deploys tags to the testing or acceptance environment by default.

git tag v1.0.0
git push origin v1.0.0

After the tag is pushed, GitHub Actions starts deployment to the corresponding validation environment. If the preview shows changes to connections, tasks, or APIs, approve the deploy gate before the import continues.

When deployment finishes, complete business validation in the testing or acceptance environment. Check functional correctness, data volume, synchronization latency, and other acceptance criteria. If validation passes, continue to production deployment.

Step 6: Trigger production deployment manually​

After testing or acceptance validation passes, deploy the same tag to production. Using the same tag ensures that production receives the exact configuration that passed validation.

The official tenant template does not include prod in the manual deployment options by default. To deploy to production, add prod to the tenant workflow options first.

  1. In the GitHub tenant repository, go to Actions, and select TapData Deploy.
  2. Click Run workflow. For Branch, select the tag, such as v1.0.0. For Target environment, select prod.
  3. Click Run workflow. If the preview shows changes to connections, tasks, or APIs, approve the deploy gate on the Actions page.
  4. After deployment finishes, sign in to the target TapData environment. Verify task status and API availability, then complete the production release.

Roll back a deployment​

If a deployment does not behave as expected, such as when a task enters an abnormal state, roll the environment back to a previous stable tag.

  1. In the GitHub tenant repository, go to Actions, and select TapData Rollback.
  2. Click Run workflow. Enter the target environment, such as prod, and the tag to roll back to, such as v0.9.0.
  3. The rollback workflow stops the current tasks, unpublishes active APIs, clears the existing configuration, imports the configuration from the selected tag, and restores the runtime state for tasks and APIs that were active before rollback.
  4. After rollback finishes, sign in to the target environment and verify the task status and API availability.

Rollback affects only the selected target environment. Other environments are not changed.

Serving Indexes and database names during rollback
  • Indexes are add-only: Rollback restores task and API configuration, but it never deletes indexes that were created in the target database. This prevents accidental removal of indexes that can affect production performance. After rollback, unused indexes appear as orphan indexes in the next deployment preview. Have a DBA review and remove them manually when appropriate.
  • DSN database names use the current Variable value: When you use DSN-based connection credentials, rollback restores project configuration and deployment logic, but the database name remains controlled by the current _DSN value in the GitHub Environment. Rolling back a Git tag does not change it automatically.

FAQ​

Q: How does project import handle existing resources?

Both automated deployment and manual import update existing connections, tasks, and APIs in the target environment based on the imported configuration. Unchanged resources remain unchanged. Resources that do not exist in the target environment are created.

With GitHub-based deployment, real connection addresses, usernames, and passwords for new connections are injected from the target Environment Secrets and Variables based on connection names. With manual import, credentials are not injected automatically. Update or complete the connection values in the target environment after import.

Q: Why does GitHub Actions report Could not find reusable workflow?

  • Check whether the Worker repository visibility is Internal.
  • Check whether the Worker repository path in the tenant workflow has been replaced with the real value.

Q: The deployment succeeded, but the database password was not injected. What should I check?

  • Check whether connection credentials are configured in the target Environment Secrets or Variables, not repository-level Secrets.
  • Check whether the variable names exactly match the connection names in TapData after applying the naming rules. When credentials are configured under an Environment, do not add the environment prefix to the credential names again.
  • Check whether the TapData connection name contains only letters, numbers, and underscores, and starts with a letter or underscore. If the connection name contains hyphens, spaces, or Chinese characters, GitHub Secrets and Variables might not be created with matching names. Rename the connection and export the project again.

Q: Git export reports git-receive-pack not permitted or cannot push. What should I check?

  • Check whether the personal access token in Git configuration has read and write permissions for Contents and Pull requests in the tenant repository.
  • For a fine-grained PAT, check whether Resource owner and Repository access include the current tenant repository.
  • If this change includes files under .github/workflows/, also make sure the token has write permission for Actions or Workflows.

Q: Import fails and reports tag does not exist. What should I do?

The target environment might be missing a tag, Agent, or another runtime resource referenced by a task in the source environment. Create a resource with the same name in the target environment, or remove bindings that are not suitable for cross-environment deployment from the source environment and export again.

Q: What should I do if the import script fails?

  • Check whether {ENV}_TAPDATA_ACCESS_CODE is configured correctly and is still valid.
  • Open the GitHub Actions logs, find the error returned by the TapData API, and troubleshoot based on that response.

Q: Why is an API marked as updated with (serving-index declarations only)?

Serving Index declarations are stored in the API configuration. Selecting or clearing an index declaration updates the API metadata even when the API path, request and response fields, and API contract do not change. The pipeline uses this label to distinguish an index-only update from an API contract change.

Q: How should I handle orphan indexes in the deployment preview?

An orphan index exists in the target collection but is not declared by any API. Orphan indexes can result from previous versions or rollbacks.

  • Impact: Unused indexes increase write overhead and count toward MongoDB's limit of 64 indexes per collection.
  • What to do: TapData does not delete existing indexes during deployment or rollback. Have an operations team member or DBA review the orphan index list and remove an index from the database only after confirming that no other workload uses it, for example with db.collection.dropIndex(...).

Appendix: Manually import configuration​

Use manual import when GitHub integration is not configured or when you need to import a specific exported version directly into a target environment.

  1. On the Advanced Settings > Export/Import page, click Import.

  2. Upload the archive exported from the development environment.

  3. Select a conflict handling policy, such as Skip or Update existing configuration.

  4. Click OK. TapData validates the file format and shows an import preview, including the affected connections, tasks, and APIs. The following image shows the import preview page, which is used to confirm the resources to be added or updated.

    View project import preview

  5. Review the preview and run the import.

  6. After import finishes, sign in to the target TapData environment. Update the real connection addresses, usernames, and passwords, test the database connections, verify task status and API availability, and then start the tasks.