Deploy HA TapData Enterprise (2 Nodes)
To ensure reliability in production environments, we recommend deploying TapData in a high-availability setup. This guide will show you how to quickly deploy a high-availability TapData service on a local Linux platform.
Software & Hardware Requirementsβ
- CPU: 8 cores
- Memory: 16 GB
- Storage Space: 100 GB
- Operating System: CentOS 7+ or Ubuntu 16.04+
Deployment Architectureβ
In this example, assume we have two servers (A and B), each configured with an IP. We aim to deploy the complete TapData service, including the management service, sync governance service, and API service, on both servers to achieve high availability.
In this environment, we have already deployed a MongoDB replica set, which provides storage services for running data for TapData services.
This guide uses CentOS 7 as an example to demonstrate the deployment process for Servers A and B.
Prerequisitesβ
Before deployment, we need to perform the following operations on both servers.
Log into the server and execute the following commands to adjust file access numbers, firewall, and other system parameters.
ulimit -n 1024000
echo "* soft nofile 1024000" >> /etc/security/limits.conf
echo "* hard nofile 1024000" >> /etc/security/limits.conf
systemctl disable firewalld.service
systemctl stop firewalld.service
setenforce 0
sed -i "s/enforcing/disabled/g" /etc/selinux/configInstall environment dependencies.
Execute the following command to install Java 1.8 version.
yum -y install java-1.8.0-openjdk
Install MongoDB (version 4.0 or above) as an intermediary database to store task data and more. For detailed steps, see the official documentation.
Download the TapData installation package (you can contact us to obtain it), and upload it to the device to be deployed.
On the device to be deployed, execute the following command to unzip the installation package and enter the unzipped path.
tar -zxvf installation_package_name && cd tapdata
Example:
tar -zxvf tapdata-release-v2.14.tar.gz && cd tapdata
Server A Deployment Processβ
Obtain the License file.
Execute the following command to obtain the required SID information for application.
java -cp components/tm.jar -Dloader.main=com.tapdata.tm.license.util.SidGenerator org.springframework.boot.loader.PropertiesLauncher
Provide the printed SID information to the TapData support team to complete the License application process.
Upload the obtained License file to the unzipped directory (tapdata).
In the tapdata directory, execute
./tapdata start
, following command line prompts to set TapData's login address, API service port, MongoDB connection information, etc., as shown below:tipIf deploying with a non-root user, avoid using
sudo
to elevate privileges to prevent installation failure. Before executing commands, usesudo chown -R <your-username>:<your-group> <installation-dir>
orsudo chmod -R 777 <installation-dir>
to grant full permissions to the installation directory for the current user../tapdata start
_______ _____ _____ _______
|__ __|/\ | __ \| __ \ /\|__ __|/\
| | / \ | |__) | | | | / \ | | / \
| | / /\ \ | ___/| | | |/ /\ \ | | / /\ \
| |/ ____ \| | | |__| / ____ \| |/ ____ \
|_/_/ \_\_| |_____/_/ \_\_/_/ \_\
WORK DIR:/root/tapdata
Init tapdata...
β Please enter backend url, comma separated list. e.g.:http://127.0.0.1:3030/ (Default: http://127.0.0.1:3030/): β¦
β Please enter tapdata port. (Default: 3030): β¦
β Please enter api server port. (Default: 3080): β¦
β Does MongoDB require username/password?(y/n): β¦ no
β Does MongoDB require TLS/SSL?(y/n): β¦ no
β Please enter MongoDB host, port, database name(Default: 127.0.0.1:27017/tapdata): β¦
β Does API Server response error code?(y/n): β¦ yes
MongoDB uri: mongodb://127.0.0.1:27017/tapdata
MongoDB connection command: mongo mongodb://127.0.0.1:27017/tapdata
System initialized. To start Tapdata, run: tapdata start
WORK DIR:/root/tapdata
Testing JDK...
java version:1.8
Java environment OK.
Unpack the files...
Restart TapdataAgent ...:
TapdataAgent starting ...:- Please enter backend url: Set the TapData platform login address, default is
http://127.0.0.1:3030/
- Please enter tapdata port: Set the TapData platform login port, default is
3030
. - Please enter api server port: Set the API Server service port, default is
3080
. - Does MongoDB require username/password?: If MongoDB database has not enabled security authentication, enter n; if enabled, enter y and follow prompts to enter username, password, and authentication database (default is
admin
). - Does MongoDB require TLS/SSL?(y/n): If MongoDB database has not enabled TSL/SSL encryption, enter n; if enabled, enter y and follow prompts to enter the absolute path of CA certificate and Certificate Key files, as well as the file password for the Certificate Key.
- Please enter MongoDB host, port, database name: Set the MongoDB database URI connection information, default is
127.0.0.1:27017/tapdata
. - Does API Server response error code?: Whether to enable the API Server response error code feature.
After successful deployment, the command line returns an example as follows:
deployed connector.
Waiting for the flow engine to start \
FlowEngine is startup at : 2023-04-01 23:00
API service started- Please enter backend url: Set the TapData platform login address, default is
Log into the TapData platform through a browser; the local login address is http://127.0.0.1:3030. Change the password promptly after the first login for security.
tipTo access the TapData service from other devices on the same internal network, ensure network interconnectivity.
Set TapData service to start on boot.
Navigate to the
/usr/lib/systemd/system
directory and use a text editor (e.g.,vim
) to create a new service file namedtapdata.service
. Paste the following content into the file.# Replace ExecStart and ExecStop paths with the correct tapdata installation path
[Unit]
Description=TapData Service
After=network.target
[Service]
Type=simple
User=root
ExecStart=/root/tapdata/tapdata start
ExecStop=/root/tapdata/tapdata stop
Restart=on-failure
[Install]
WantedBy=multi-user.targetLoad the new service file and enable it to start on boot:
sudo systemctl daemon-reload
sudo systemctl enable tapdata.service(Optional) Restart the machine during off-peak hours and check if the TapData service started normally with the
systemctl status tapdata.service
command.
Server B Deployment Processβ
Obtain the License file.
Execute the following command to obtain the required SID information for application.
java -cp components/tm.jar -Dloader.main=com.tapdata.tm.license.util.SidGenerator org.springframework.boot.loader.PropertiesLauncher
Provide the printed SID information to the TapData support team to complete the License application process.
Upload the obtained License file to the unzipped directory (tapdata).
In the tapdata directory, execute
./tapdata start
, following command line prompts to set TapData's login address, API service port, MongoDB connection information, etc., as shown below:tipIf deploying with a non-root user, avoid using
sudo
to elevate privileges to prevent installation failure. Before executing commands, usesudo chown -R <your-username>:<your-group> <installation-dir>
orsudo chmod -R 777 <installation-dir>
to grant full permissions to the installation directory for the current user../tapdata start
_______ _____ _____ _______
|__ __|/\ | __ \| __ \ /\|__ __|/\
| | / \ | |__) | | | | / \ | | / \
| | / /\ \ | ___/| | | |/ /\ \ | | / /\ \
| |/ ____ \| | | |__| / ____ \| |/ ____ \
|_/_/ \_\_| |_____/_/ \_\_/_/ \_\
WORK DIR:/root/tapdata
Init tapdata...
β Please enter backend url, comma separated list. e.g.:http://127.0.0.1:3030/ (Default: http://127.0.0.1:3030/): β¦ http://192.168.1.200:3030,http://192.168.1.201:3030
β Please enter tapdata port. (Default: 3030): β¦
β Please enter api server port. (Default: 3080): β¦
β Does MongoDB require username/password?(y/n): β¦ no
β Does MongoDB require TLS/SSL?(y/n): β¦ no
β Please enter MongoDB host, port, database name(Default: 127.0.0.1:27017/tapdata): β¦ 192.168.1.200:27017/tapdata
β Does API Server response error code?(y/n): β¦ yes
MongoDB uri: mongodb://192.168.1.200:27017/tapdata
MongoDB connection command: mongo mongodb://192.168.1.200:27017/tapdata
System initialized. To start Tapdata, run: tapdata start
WORK DIR:/root/tapdata
Testing JDK...
java version:1.8
Java environment OK.
Unpack the files...
frontend server started.begin deploy init
Try to connect to TM for deploy connector...
deploy connector...Please enter backend url: Set the TapData platform login addresses for both Servers A and B, separated by a comma, e.g.,
http://192.168.1.200:3030,http://192.168.1.201:3030
.Please enter tapdata port: Set the TapData platform login port, default is
3030
.Please enter api server port: Set the API Server service port, default is
3080
.Does MongoDB require username/password?: If MongoDB database has not enabled security authentication, enter n; if enabled, enter y and follow prompts to enter username, password, and authentication database (default is
admin
).Does MongoDB require TLS/SSL?(y/n): If MongoDB database has not enabled TSL/SSL encryption, enter n; if enabled, enter y and follow prompts to enter the absolute path of CA certificate and Certificate Key files, as well as the file password for the Certificate Key.
Please enter MongoDB host, port, database name: Set the MongoDB database URI connection information, in this case,
mongodb://192.168.1.200:27017/tapdata
.Does API Server response error code?: Whether to enable the API Server response error code feature.
After successful deployment, the command line returns an example as follows:
deployed connector.
Waiting for the flow engine to start \
FlowEngine is startup at : 2023-04-01 23:10
API service started
With TapData services deployed on both servers, devices on the same internal network can access the management page via http://192.168.1.200:3030 or http://192.168.1.201:3030.
tipChange the password promptly after the first login for security.
After successful login, in System Management > Cluster Management, you can view the status of TapData services on both servers.
Set TapData service to start on boot.
Navigate to the
/usr/lib/systemd/system
directory and use a text editor (e.g.,vim
) to create a new service file namedtapdata.service
. Paste the following content into the file.# Ensure ExecStart and ExecStop point to the correct tapdata installation path
[Unit]
Description=TapData Service
After=network.target
[Service]
Type=simple
User=root
ExecStart=/root/tapdata/tapdata start
ExecStop=/root/tapdata/tapdata stop
Restart=on-failure
[Install]
WantedBy=multi-user.targetLoad the new service file and enable it to start on boot:
sudo systemctl daemon-reload
sudo systemctl enable tapdata.service(Optional) Restart the machine during off-peak hours and check if the TapData service started normally with the
systemctl status tapdata.service
command.