Skip to main content

Deploy HA TapData Enterprise (2 Nodes)

Applicable EditionsTapData EnterpriseTapData Enterprise can be deployed in your local data center, making it suitable for scenarios with strict requirements on data sensitivity or network isolation. It can serve to build real-time data warehouses, enable real-time data exchange, data migration, and more.

To ensure reliability in production environments, we recommend deploying TapData in a high-availability setup. This guide will show you how to quickly deploy a high-availability TapData service on a local Linux platform.

Software & Hardware Requirements​

  • CPU: 8 cores
  • Memory: 16 GB
  • Storage Space: 100 GB
  • Operating System: CentOS 7+ or Ubuntu 16.04+

Deployment Architecture​

In this example, assume we have two servers (A and B), each configured with an IP. We aim to deploy the complete TapData service, including the management service, sync governance service, and API service, on both servers to achieve high availability.

tip

In this environment, we have already deployed a MongoDB replica set, which provides storage services for running data for TapData services.

Deployment Architecture

This guide uses CentOS 7 as an example to demonstrate the deployment process for Servers A and B.

Prerequisites​

Before deployment, we need to perform the following operations on both servers.

  1. Log into the server and execute the following commands to adjust file access numbers, firewall, and other system parameters.

    ulimit -n 1024000 
    echo "* soft nofile 1024000" >> /etc/security/limits.conf 
    echo "* hard nofile 1024000" >> /etc/security/limits.conf 
    systemctl disable firewalld.service 
    systemctl stop firewalld.service 
    setenforce 0 
    sed -i "s/enforcing/disabled/g" /etc/selinux/config

  2. Install environment dependencies.

    1. Execute the following command to install Java 1.8 version.

      yum -y install java-1.8.0-openjdk

    2. Install MongoDB (version 4.0 or above) as an intermediary database to store task data and more. For detailed steps, see the official documentation.

  3. Download the TapData installation package (you can contact us to obtain it), and upload it to the device to be deployed.

  4. On the device to be deployed, execute the following command to unzip the installation package and enter the unzipped path.

    tar -zxvf installation_package_name && cd tapdata

    Example: tar -zxvf tapdata-release-v2.14.tar.gz && cd tapdata

Server A Deployment Process​

  1. Obtain the License file.

    1. Execute the following command to obtain the required SID information for application.

      java -cp components/tm.jar -Dloader.main=com.tapdata.tm.license.util.SidGenerator org.springframework.boot.loader.PropertiesLauncher

    2. Provide the printed SID information to the TapData support team to complete the License application process.

    3. Upload the obtained License file to the unzipped directory (tapdata).

  2. In the tapdata directory, execute ./tapdata start, following command line prompts to set TapData's login address, API service port, MongoDB connection information, etc., as shown below:

    tip

    If deploying with a non-root user, avoid using sudo to elevate privileges to prevent installation failure. Before executing commands, use sudo chown -R <your-username>:<your-group> <installation-dir> or sudo chmod -R 777 <installation-dir> to grant full permissions to the installation directory for the current user.

     ./tapdata start
     _______       _____  _____       _______
    |__   __|/\   |  __ \|  __ \   /\|__   __|/\    
       | |  /  \  | |__) | |  | | /  \  | |  /  \   
       | | / /\ \ |  ___/| |  | |/ /\ \ | | / /\ \  
       | |/ ____ \| |    | |__| / ____ \| |/ ____ \ 
       |_/_/    \_\_|    |_____/_/    \_\_/_/    \_\ 

    WORK DIR:/root/tapdata
    Init tapdata...
    ✔ Please enter backend url, comma separated list. e.g.:http://127.0.0.1:3030/ (Default: http://127.0.0.1:3030/):  …
    ✔ Please enter tapdata port. (Default: 3030):  …
    ✔ Please enter api server port. (Default: 3080):  …
    ✔ Does MongoDB require username/password?(y/n):  … no
    ✔ Does MongoDB require TLS/SSL?(y/n):  … no
    ✔ Please enter MongoDB host, port, database name(Default: 127.0.0.1:27017/tapdata):  …
    ✔ Does API Server response error code?(y/n):  … yes
    MongoDB uri:  mongodb://127.0.0.1:27017/tapdata
    MongoDB connection command: mongo  mongodb://127.0.0.1:27017/tapdata
    System initialized. To start Tapdata, run: tapdata start
    WORK DIR:/root/tapdata
    Testing JDK...
    java version:1.8
    Java environment OK.
    Unpack the files...
    Restart TapdataAgent ...:
    TapdataAgent starting ...:
    • Please enter backend url: Set the TapData platform login address, default is http://127.0.0.1:3030/
    • Please enter tapdata port: Set the TapData platform login port, default is 3030.
    • Please enter api server port: Set the API Server service port, default is 3080.
    • Does MongoDB require username/password?: If MongoDB database has not enabled security authentication, enter n; if enabled, enter y and follow prompts to enter username, password, and authentication database (default is admin).
    • Does MongoDB require TLS/SSL?(y/n): If MongoDB database has not enabled TSL/SSL encryption, enter n; if enabled, enter y and follow prompts to enter the absolute path of CA certificate and Certificate Key files, as well as the file password for the Certificate Key.
    • Please enter MongoDB host, port, database name: Set the MongoDB database URI connection information, default is 127.0.0.1:27017/tapdata.
    • Does API Server response error code?: Whether to enable the API Server response error code feature.

    After successful deployment, the command line returns an example as follows:

    deployed connector.
    Waiting for the flow engine to start \
    FlowEngine is startup at : 2023-04-01 23:00
    API service started

  1. Log into the TapData platform through a browser; the local login address is http://127.0.0.1:3030. Change the password promptly after the first login for security.

    tip

    To access the TapData service from other devices on the same internal network, ensure network interconnectivity.

  2. Set TapData service to start on boot.

    1. Navigate to the /usr/lib/systemd/system directory and use a text editor (e.g., vim) to create a new service file named tapdata.service. Paste the following content into the file.

      # Replace ExecStart and ExecStop paths with the correct tapdata installation path
      [Unit]
      Description=TapData Service
      After=network.target

      [Service]
      Type=simple
      User=root
      ExecStart=/root/tapdata/tapdata start
      ExecStop=/root/tapdata/tapdata stop
      Restart=on-failure

      [Install]
      WantedBy=multi-user.target

    2. Load the new service file and enable it to start on boot:

      sudo systemctl daemon-reload
      sudo systemctl enable tapdata.service

    3. (Optional) Restart the machine during off-peak hours and check if the TapData service started normally with the systemctl status tapdata.service command.

Server B Deployment Process​

  1. Obtain the License file.

    1. Execute the following command to obtain the required SID information for application.

      java -cp components/tm.jar -Dloader.main=com.tapdata.tm.license.util.SidGenerator org.springframework.boot.loader.PropertiesLauncher

    2. Provide the printed SID information to the TapData support team to complete the License application process.

    3. Upload the obtained License file to the unzipped directory (tapdata).

  2. In the tapdata directory, execute ./tapdata start, following command line prompts to set TapData's login address, API service port, MongoDB connection information, etc., as shown below:

    tip

    If deploying with a non-root user, avoid using sudo to elevate privileges to prevent installation failure. Before executing commands, use sudo chown -R <your-username>:<your-group> <installation-dir> or sudo chmod -R 777 <installation-dir> to grant full permissions to the installation directory for the current user.

    ./tapdata start
     _______       _____  _____       _______
    |__   __|/\   |  __ \|  __ \   /\|__   __|/\    
       | |  /  \  | |__) | |  | | /  \  | |  /  \   
       | | / /\ \ |  ___/| |  | |/ /\ \ | | / /\ \  
       | |/ ____ \| |    | |__| / ____ \| |/ ____ \ 
       |_/_/    \_\_|    |_____/_/    \_\_/_/    \_\ 

    WORK DIR:/root/tapdata
    Init tapdata...
    ✔ Please enter backend url, comma separated list. e.g.:http://127.0.0.1:3030/ (Default: http://127.0.0.1:3030/):  … http://192.168.1.200:3030,http://192.168.1.201:3030
    ✔ Please enter tapdata port. (Default: 3030):  … 
    ✔ Please enter api server port. (Default: 3080):  … 
    ✔ Does MongoDB require username/password?(y/n):  … no
    ✔ Does MongoDB require TLS/SSL?(y/n):  … no
    ✔ Please enter MongoDB host, port, database name(Default: 127.0.0.1:27017/tapdata):  … 192.168.1.200:27017/tapdata
    ✔ Does API Server response error code?(y/n):  … yes
    MongoDB uri:  mongodb://192.168.1.200:27017/tapdata
    MongoDB connection command: mongo  mongodb://192.168.1.200:27017/tapdata
    System initialized. To start Tapdata, run: tapdata start
    WORK DIR:/root/tapdata
    Testing JDK...
    java version:1.8
    Java environment OK.
    Unpack the files...
    frontend server started.begin deploy init
    Try to connect to TM for deploy connector...
    deploy connector...

    • Please enter backend url: Set the TapData platform login addresses for both Servers A and B, separated by a comma, e.g., http://192.168.1.200:3030,http://192.168.1.201:3030.

    • Please enter tapdata port: Set the TapData platform login port, default is 3030.

    • Please enter api server port: Set the API Server service port, default is 3080.

    • Does MongoDB require username/password?: If MongoDB database has not enabled security authentication, enter n; if enabled, enter y and follow prompts to enter username, password, and authentication database (default is admin).

    • Does MongoDB require TLS/SSL?(y/n): If MongoDB database has not enabled TSL/SSL encryption, enter n; if enabled, enter y and follow prompts to enter the absolute path of CA certificate and Certificate Key files, as well as the file password for the Certificate Key.

    • Please enter MongoDB host, port, database name: Set the MongoDB database URI connection information, in this case, mongodb://192.168.1.200:27017/tapdata.

    • Does API Server response error code?: Whether to enable the API Server response error code feature.

      After successful deployment, the command line returns an example as follows:

      deployed connector.
      Waiting for the flow engine to start \
      FlowEngine is startup at : 2023-04-01 23:10
      API service started

  3. With TapData services deployed on both servers, devices on the same internal network can access the management page via http://192.168.1.200:3030 or http://192.168.1.201:3030.

    tip

    Change the password promptly after the first login for security.

    After successful login, in System Management > Cluster Management, you can view the status of TapData services on both servers.

    Cluster Status

  4. Set TapData service to start on boot.

    1. Navigate to the /usr/lib/systemd/system directory and use a text editor (e.g., vim) to create a new service file named tapdata.service. Paste the following content into the file.

      # Ensure ExecStart and ExecStop point to the correct tapdata installation path
      [Unit]
      Description=TapData Service
      After=network.target

      [Service]
      Type=simple
      User=root
      ExecStart=/root/tapdata/tapdata start
      ExecStop=/root/tapdata/tapdata stop
      Restart=on-failure

      [Install]
      WantedBy=multi-user.target

    2. Load the new service file and enable it to start on boot:

      sudo systemctl daemon-reload
      sudo systemctl enable tapdata.service

    3. (Optional) Restart the machine during off-peak hours and check if the TapData service started normally with the systemctl status tapdata.service command.

Next Steps​

Connect to a Database