Skip to main content

What's New

This article provides release notes for TapData Enterprise, including new features, improvements, and bug fixes.

4.16.0​

New Features​

  • When create an API, you can now configure data masking rules for specific parameters. After enabling, those parameters are displayed as masked values in API audit logs, improving sensitive data protection.
  • You can now build and manage logical views in the Master Data Model (MDM) using MongoDB Aggregation Pipelines.

Enhancements​

  • Removed the obsolete alert options "When a validation task fails" and "When the data source network is unreachable" to simplify alert configuration.
  • Optimized MongoDB-to-SQL Server synchronization to support writing Object and Array values directly into Text fields.

Bug Fixes​

  • Fixed an error in API service debugging caused by older MongoDB versions not supporting allowDiskUse.
  • Fixed slow queries on API-related tables in the TapData intermediate database to improve system responsiveness.
  • Fixed an issue where property types within array objects were incorrectly converted to arrays in API Server responses.
  • Fixed an issue where data synchronization could be blocked when MongoDB encountered transaction errors.
  • Fixed a potential out-of-memory issue when starting MongoDB mining tasks.
  • Fixed an issue where single-node Oracle tasks could fail to start from a specified incremental timestamp due to abnormal log file matching.
  • Fixed potential data inconsistencies when syncing Blob/Clob fields from Oracle to MySQL.
  • Fixed a task failure when syncing from PostgreSQL to a sharded MongoDB collection due to shard key updates.
  • Fixed an issue where some update operations could cause other columns to become empty when syncing from PostgreSQL to StarRocks.
  • Fixed an issue where StarRocks targets could not correctly advance checkpoints in specific scenarios.
  • Fixed a formatting issue in the debugging panel when the last line of Advanced JS node code was a comment.
  • Fixed an issue where re-adding nodes after configuring an incremental start time caused the start time setting to be reset.
  • Fixed an issue where the data model comparison feature could not enable the ignore-case option.
  • Fixed an issue with the permission assignment and override logic in role management.
  • Fixed an issue where incremental tasks unexpectedly re-ran full loads after rebuilding cache.
  • Fixed inconsistent input/output change statistics in shared mining tasks.
  • Fixed an issue where Kafka-to-StarRocks sync tasks could not automatically re-authenticate after an AD server restart.
  • Fixed an issue where some connectors required manual re-registration after upgrading to version 3.27.

4.15.0​

New Features​

  • The Paimon connector now supports per-table partitioning configuration, allowing customization of partition keys, buckets, compression formats, and other attributes to improve data management flexibility.

Enhancements​

  • Significantly improved the data writing performance of the Paimon connector under multiple update conditions.
  • The StarRocks connector now adapts to High Availability (HA) architectures, ensuring continuous and stable data writing.
  • Added WSS protocol support for the communication port (8246) between the engine and the management node, enhancing communication security.

Bug Fixes​

  • Fixed an issue where AVRO formatted data with Debezium structure in Kafka could not be parsed correctly.
  • Fixed an issue where the MariaDB to Doris synchronization task failed during the CDC phase after the full sync was completed.
  • Fixed a synchronization failure in MySQL tasks caused by a field rename processor exception.
  • Fixed a task execution error caused by BSONRegExp type data when syncing from MongoDB to MySQL.
  • Fixed an issue where PostgreSQL CDC tasks could not be recovered after an engine restart due to cleaned-up WAL logs.
  • Fixed a task error when enabling the full breakpoint resume function for TDSQL.
  • Fixed an issue where the API Server audit log still contained exception information when the request was successful (Code 200).
  • Fixed an issue where duplicate name validation was missing when creating an API.
  • Fixed an issue where published API interfaces could not correctly handle parameter filtering.
  • Fixed a failure when publishing an API based on MySQL.
  • Fixed an issue where the generated API could not query data by primary key after the _id field was deleted in the synchronization task.
  • Fixed a task startup failure when using MongoDB 8.0 as intermediate storage.
  • Fixed a performance degradation in full data batch writing after enabling multi-table concurrent reading.
  • Fixed a false alarm stating the tag did not exist when starting a task after modifying the connection's engine tag.
  • Fixed the loss of query filter status when returning to the list page from the edit page on the connection management page.
  • Fixed an issue where scheduled validation tasks abnormally triggered secondary validations in a loop.

4.14.0​

New Features​

  • When using MongoDB as a source, TapData now intelligently identifies memory overflow risks based on queue/document size and prompts users to adjust batch sizes, ensuring stable task operation.

Enhancements​

  • Enhanced JS node support for stored procedure calls, improving compatibility for complex scenarios like input parameter handling and multiple result set returns.
  • The API service now directly displays code examples for obtaining Tokens, simplifying the calling process.
  • Optimized the Paimon connector reading logic to improve data synchronization efficiency.
  • Added support for sorting task tags to improve the filtering experience.
  • Optimized the License update process and node count calculation logic.

Bug Fixes​

  • Fixed an issue where field names in the Master-Slave Merge node on the task monitoring page were truncated and could not be copied.
  • Fixed an issue in Master-Slave Merge tasks where the presence of a disabled table prevented cache rebuilding for all tables.
  • Fixed a data truncation issue caused by insufficient precision when mapping Double type fields from Db2 to PostgreSQL.
  • Fixed a character set encoding error when reading data containing special characters from Db2.
  • Fixed a potential permission validation warning for PostgreSQL partition tables during the CDC phase.
  • Fixed an issue where AS400 GRAPHIC type fields were not automatically converted to compatible types in the target database.

4.13.0​

Bug Fixes​

  • Fixed an issue where the API service token expiration time was too short.
  • Fixed an issue where the API Server did not support the sorting parameter "order": "effectiveStartDate desc".
  • Fixed an anomaly in incremental verification where the verification time matched the automatic re-verification time.
  • Fixed an issue in the Master-Slave Merge node where selecting Array as the merge type for a child table incorrectly inferred the model as Document.
  • Fixed a failure in rebuilding the cache for Master-Slave Merge tasks after creating a new external storage.
  • Fixed potential memory leaks in large-scale data synchronization scenarios, improving long-running system stability.
  • Fixed a NullPointerException during model comparison when syncing from PostgreSQL to a SQL Server partitioned table.
  • Fixed unexpected retry logic behavior when the PostgreSQL source database is unavailable.
  • Fixed an issue where replication slots were not automatically cleaned up in shared mining tasks with PostgreSQL sources, causing slot accumulation.

4.12.0​

New Features​

  • System settings now support environment-level visual labels. When enabled, an environment tag is shown in the top bar to help you quickly identify the deployment and reduce operational mistakes.
  • Data model comparison adds primary-key consistency checks for manually created tables, helping you detect and fix PK mismatches earlier.
  • Task monitoring adds P99/P95 latency metrics with time-range views for clearer performance and tail-latency visibility.

Enhancements​

  • Standardized naming for all data sources to improve consistency and readability across configurations.
  • Improved the API publishing experience by disabling selection and renaming of nested fields under Array fields, reducing the risk of accidental misconfiguration.
  • Replication tasks can now aggregate and report all tables with anomalies in one go while running, making issue triage faster.

Bug Fixes​

  • Fixed an issue where tasks could stop unexpectedly after adding tables dynamically in multi-table merge scenarios.

4.11.0​

New Features​

  • Released Db2 for i connector with real-time sync and write support for enterprise data integration.
  • Incremental data validation now supports bulk manual verification and repair, configurable validation scope, and per-table ignore lists.
  • The master-child merge node adds child-table cache rebuild to quickly fix cache inconsistencies and protect data quality.
  • Data replication tasks extend exactly-once writes with MongoDB and Sybase as supported targets, ensuring idempotent and consistent writes.
  • Task logs add DDL output for easier troubleshooting and audit reviews.
  • Incremental reads are now auto-tuned by adjusting batch size to balance throughput and stability.

Enhancements​

  • API audit logs now include database response time.
  • Improved API performance for faster overall request handling.
  • Improved API debug page results display for better readability and debugging.
  • Improved the logic for switching master-child merge tasks into incremental mode.

Bug Fixes​

  • Fixed a memory leak that could occur in specific scenarios.

4.10.0​

New Features​

  • Added API multi-node high-availability to improve service stability.
  • In Operational Data Hub table-detail lineage view, task lag is now displayed so you can spot pipeline delays at a glance.
  • Incremental data validation can now generate custom repair SQL, letting DBA review and apply fixes selectively.

Enhancements​

  • PostgreSQL sources without a primary key but with multiple unique indexes now auto-pick one as the replica-identity key (shown as “coreUnique” in the UI).
  • PostgreSQL connector lets you split a primary-key update into separate DELETE + INSERT events.

4.9.0​

New Features​

Enhancements​

  • While creating APIs, parameters now support the IN operator (e.g., caseType IN ('I', 'A')). Custom queries now support required/optional parameter settings and field filtering.
  • Optimized QPS metrics display on the task monitoring page, now including CPU and memory usage trend charts.
  • UI enhancements across the platform for a better user experience.

Bug Fixes​

  • Fixed an issue where setting group tags in user management could incorrectly set the user status to “Reject.”
  • Fixed a problem in multi-node deployments where license updates on the frontend were not propagated to all nodes.

4.8.0​

New Features​

Enhancements​

  • Enabled rolling upgrades for the API service to ensure uninterrupted availability during updates.

Bug Fixes​

  • Fixed an issue where the audit log could not identify client information when the token had expired.
  • Fixed an issue where the Limit parameter in API requests was not being applied correctly.

4.7.0​

New Features​

  • Added support for configuring field masking rules when creating APIs. Masked fields will also appear masked in audit logs and debug results, improving data security.
  • When publishing custom query APIs, users can now preview and edit the final query statement and use custom parameter values.
  • API responses now support returning only specific fields from nested documents or arrays, reducing data redundancy and improving performance.

Enhancements​

  • Added the ability to configure API database timeout settings in the system settings. When a request times out, clear log messages are generated to simplify troubleshooting.
  • Improved the generated API Swagger documentation to fully include request bodies, response formats, and error codes, enhancing integration efficiency and consistency.

4.6.0​

New Features​

  • Added support for defining business aliases for fields when creating APIs, improving semantic readability and standardization of field names.
  • When publishing a Custom Query API, you can now preview and edit the generated SQL statement, and reference user-defined parameters—offering a more flexible API configuration experience.

Enhancements​

  • Added an “Ignore Update Events” option for child table merge logic, reducing redundant triggers and minimizing unnecessary data updates.
  • Support for customizing license alert levels and email templates, helping enterprises meet compliance and notification requirements.
  • Added a Set API Access option on the role management page, enabling quick assignment of API access permissions to roles, improving authorization efficiency.

Bug Fixes​

  • Fixed an issue where the maximum number of rows returned by an API did not take effect in system settings.
  • Fixed a display error in the API audit page.

4.5.0​

Enhancements​

  • Improved write performance when using Feishu Bitable as a target database.
  • Optimized the source and target configuration interfaces to make interactions clearer, improving usability and configuration efficiency.

Bug Fixes​

  • Fixed an issue where specified return fields for APIs were not applied correctly.

4.4.0​

New Features​

  • Incremental data validation now supports tables without primary keys, along with manual repair. This expands validation coverage and helps ensure data consistency.
  • Added alert notifications during task retries to improve visibility and responsiveness in failure scenarios.
  • Introduced "Node Data Preview" in the task editor, allowing you to view sample fields instantly while configuring tasks—speeding up design and troubleshooting.

Enhancements​

  • Improved data source monitoring: Added connection status and performance metrics for Sybase and PostgreSQL, enhancing system observability.

4.3.0​

New Features​

  • Data validation now supports exporting repair SQL for inconsistent results, with compatibility for PostgreSQL, Sybase, and SQL Server. This makes offline auditing and rollback easier for ops teams.
  • JS Processor Node now supports the unset operation, enabling targeted field removal in MongoDB. Ideal for cleanup and payload optimization.

Enhancements​

  • Refined logic for marking full sync tasks as completed: now based on whether all writes to the target are finished—more aligned with real business completion.

4.2.0​

New Features​

Enhancements​

  • Optimized index synchronization logic to preserve index names in the target database, improving consistency and business traceability.

Bug Fixes​

  • Fixed an issue where searching shared mining tasks by connection name caused a page error.

4.1.0​

New Features​

  • Added support for Incremental Data Validation within tasks. This feature continuously verifies target-side data consistency during synchronization, improving validation efficiency and enhancing overall data reliability.

Enhancements​

  • Improved the user experience of the data validation interface to make the process more intuitive and feedback more informative.

Bug Fixes​

  • Fixed an issue where an unclear error message was shown when creating an Oracle data source with a non-existent account. The system now explicitly indicates that the account does not exist.

4.0.0​

New Features​

  • Introduced Tapdata MCP (Model Context Protocol), enabling integration of multi-source data into real-time contextual views consumable by LLMs and AI Agents. This feature is ideal for scenarios with high demands on data freshness and compliance, such as financial risk control.
  • Added support for using StarRocks as a target database, allowing faster construction of real-time data warehouses for high-concurrency, multi-dimensional analytics use cases.
  • Added the ability to choose from multiple data structures(e.g. Flink) when syncing to Kafka-Enhanced, enhancing compatibility and integration efficiency with downstream systems.

Enhancements​

  • When update or delete operations fail to match records on the target side, the system now logs warning messages to aid troubleshooting.
  • Data replication tasks now support filtering tables by primary keys and unique indexes, including: primary key only, unique index only, with either, or without primary key (optionally including unique index).
  • In MySQL-to-MySQL sync scenarios, you can now create indexes based on the first N characters of a text-type field.

Bug Fixes​

  • Fixed an issue where field and table processing nodes displayed incorrectly across pagination in replication tasks.
  • Fixed an ArrayIndexOutOfBoundsException that could occur during incremental sync from Oracle to Doris.
  • Fixed an error that occurred when using a date field as a join key for data validation during sync between Oracle databases.