Skip to main content

KingbaseES-R6

Applicable EditionsTapData CloudTapData Cloud offers you cloud services that are suitable for scenarios requiring rapid deployment and low initial investment, helping you focus more on business development rather than infrastructure management. Free trial with TapData Cloud.TapData CommunityTapData Community is an open-source data integration platform that provides basic data synchronization and transformation capabilities. This helps you quickly explore and implement data integration projects. As your project or business grows, you can seamlessly upgrade to TapData Cloud or TapData Enterprise to access more advanced features and service support.

The Kingbase Database Management System (KingbaseES) is a commercial relational database management system developed independently by Beijing Kingbase Technology Inc, with proprietary intellectual property rights. KingbaseES-R6 is compatible with most features of Postgres 9.6 version. This article will introduce how to add KingbaseES-R6 data source in TapData Cloud, which can then be used as a source or target database to build data pipelines.

Supported Versions​

KingBaseES-V8R6

tip

The database modes supported by KingbaseES-R6 are Oracle, PostgreSQL, and MySQL. It should be noted that in Oracle mode, objects are lowercase by default. For more information, see Kingbase ES Official Documentation.

Maturity Stage​

Beta Data Source
The Beta Data Source is currently in public preview and has undergone thorough testing, including basic test cases and integration test cases. However, it has not yet completed the TapData certification test process.

Prerequisites​

  1. Log in to the KingbaseES-R6 database as an administrator.

  2. Execute the following commands to create an account for data synchronization/development tasks.

    CREATE USER username WITH PASSWORD 'password';
    • username: Username.
    • password: Password.

  3. Grant permissions to the newly created account as required or customize permission control based on business needs.

-- Enter the database for which you want to grant permissions
\c database_name

-- Grant SELECT permission on all tables in the source schema
GRANT SELECT ON ALL TABLES IN SCHEMA schema_name TO username;

-- Grant USAGE permission on the source schema
GRANT USAGE ON SCHEMA schema_name TO username;
  • database_name: Database name.
  • schema_name: Schema name.
  • username: Username.

  1. To capture incremental changes from the source database, we need to execute the following command to set the replication identity to FULL (using the entire row as the identifier). This property determines the fields recorded in the log when UPDATE/DELETE operations occur.

    ALTER TABLE schema_name.table_name REPLICA IDENTITY FULL;
    • schema_name: The name of the schema.
    • table_name: The name of the table.

    After completing this operation, you will also need to contact technical support to provide the relevant plugin for installation on the server hosting KingbaseES-R6.

Connect to KingbaseES-R6​

  1. Log in to TapData Platform.

  2. In the left navigation panel, click Connections.

  3. On the right side of the page, click Create.

  4. In the pop-up dialog, select KingbaseES-R6.

  5. On the page that you are redirected to, follow the instructions below to fill in the connection information for KingbaseES-R6.

    KingbaseES-R6 Connection Example

    • Connection Information Settings
      • Name: Fill in a unique name that has business significance.
      • Type: Supports KingbaseES-R6 as a source or target database.
      • Host: The database connection address.
      • Port: The service port of database.
      • Database: Database name, a connection corresponding to a database, if there are multiple databases, you need to create multiple connections.
      • Schema: Schema name.
      • ExtParams: Additional connection parameters, default empty.
      • User: The database username.
      • Password: The database password.
      • Log Plugin Name: To read the data changes of KingbaseES-R6 and achieve incremental data synchronization, you need to complete the installation of the plugin according to the guidance of the preparations.
    • Advanced Settings
      • Timezone: Defaults to the time zone used by the database, which you can also manually specify according to your business needs.
      • CDC Log Caching: Mining the source database's incremental logs, this feature allows multiple tasks to share incremental logs from the source database, avoiding redundant reads and thus significantly reducing the load on the source database during incremental synchronization. Upon enabling this feature, an external storage should be selected to store the incremental log.
      • Contain Table: The default option is All, which includes all tables. Alternatively, you can select Custom and manually specify the desired tables by separating their names with commas (,).
      • Exclude Tables: Once the switch is enabled, you have the option to specify tables to be excluded. You can do this by listing the table names separated by commas (,) in case there are multiple tables to be excluded.
      • Agent Settings: Defaults to Platform automatic allocation, you can also manually specify an agent.
      • Model Load Time: If there are less than 10,000 models in the data source, their information will be updated every hour. But if the number of models exceeds 10,000, the refresh will take place daily at the time you have specified.
      • Enable Heartbeat Table: When the connection type is selected as Source and Target or Source, you can enable this option to create a heartbeat table named _tapdata_heartbeat_table in the source database. It will be updated every 10 seconds by TapData (requires relevant permissions) and used for monitoring the health of the data source connection and tasks.

  6. Click Connection Test, and when passed, click Save.

    tip

    If the connection test fails, follow the prompts on the page to fix it.