Skip to main content

Enable Real-Time Data Hub

Applicable EditionsTapData CloudTapData Cloud offers you cloud services that are suitable for scenarios requiring rapid deployment and low initial investment, helping you focus more on business development rather than infrastructure management. Free trial with TapData Cloud.TapData CommunityTapData Community is an open-source data integration platform that provides basic data synchronization and transformation capabilities. This helps you quickly explore and implement data integration projects. As your project or business grows, you can seamlessly upgrade to TapData Cloud or TapData Enterprise to access more advanced features and service support.

Due to digital transformation, the presence of isolated data, data fragmentation, or data silos has emerged as a significant challenge. Moreover, there is a growing demand for data in business operations. However, traditional data delivery methods pose limitations, such as lengthy processes and substantial resource requirements. This situation calls for a solution that enables organizations to swiftly establish data flow pipelines and unlock the value of their data.

TapData Cloud's Real-Time Data Hub offers a powerful solution. By synchronizing data from diverse business systems to a unified platform cache layer, it enables the consolidation of data sources and facilitates seamless data processing and analysis. This unified and real-time data platform helps enterprises overcome data silos and promotes data-driven decision-making, ultimately enhancing their competitiveness in the market.

Background​

In today's digital age, one of the greatest challenges for enterprises is how to efficiently process and analyze vast amounts of real-time data. Traditional methods of data handling, such as batch processing or manually writing data ETL scripts, often fail to provide timely data analysis and processing. This limitation restricts businesses' ability to make prompt decisions in a rapidly changing market environment. Moreover, performing data operations directly on production databases can impact their stability and security, affecting overall business efficiency.

The introduction of a Real-Time Data Hub aims to resolve these issues. It provides an efficient and reliable platform that helps businesses process and analyze data in real time, quickly responding to market and customer demands. For example:

  • By integrating TapData's Real-Time Data Hub, a company successfully built a data dashboard to monitor cloud-based user behavior. They streamed database data in real time to TapData’s platform cache layer, allowing real-time processing of cache layer data to generate key business metrics without affecting the source databases. This provided the freshest data for necessary BI reports, offering immediate business insights and analysis.
  • In another case, a retail enterprise utilized the Real-Time Data Hub to build a data portal. This portal enabled front-end business developers to quickly discover and process data through self-service, allowing them to build and publish APIs. Using TapData's data catalog, they could rapidly locate necessary data, enabling self-service processing and modeling. This not only enhanced development efficiency but also reduced reliance on specialized data teams, saving the enterprise substantial costs.

These cases collectively demonstrate how the Real-Time Data Center can help businesses overcome the limitations of traditional data handling, offering more efficient and flexible data management solutions. Through real-time data processing, enterprises can better grasp market dynamics, quickly respond to customer needs, and maintain a competitive edge.

Real-Time Data Hub Introduction​

With the increase in the tasks carried by the source database, in order to minimize the impact of data extraction on the source database and adhere to the organization's concept of data hierarchical governance, TapData organizes the data service platform in a layered manner based on the data flow order. This hierarchical arrangement ensures efficient and structured data processing, allowing for better data management and seamless integration across different systems.

Data Service Platform Architecture

HierarchyDescription
SourcesTapData consolidates data sources from various business systems into a centralized data source layer, which serves as the initial step in bridging data silos. This abstraction of data sources enables a unified and streamlined approach to accessing and utilizing data. For more detailed instructions, please refer to the Connect Data Sources section for comprehensive information on establishing connections with your data sources.
FDM (Foundation Data Model)By synchronizing the table from the source database to the FDM beforehand, the data can be readily accessed by the business through the FDM, thus eliminating the need to directly read or manipulate the data in the source database, such as performing union operations, during data processing. This approach significantly minimizes the impact on the business operations of the source database.
MDM (Master Data Model)If there is a need for extensive customization of data processing or operations, such as generating a wide table, it is possible to extract the data table from the FDM and perform the required operations within the MDM. This allows for the generation of model data that can be used in the final business processes.
Targets & ServiceTapData provides a centralized platform that aggregates and presents various data sources, allowing them to be utilized as targets for data processing. This enables the provision of processed data to the business, facilitating the creation of a unified data service platform for enterprises.

Procedure​

In the Real-Time Data Hub, we need to prepare a MongoDB database as a data repository for the Data Cache Layer and Data Processing Layer.

  1. Log in to TapData Platform.

  2. In the left navigation panel, click Real-Time Data Hub.

  3. Choose the steps based on your product series:

  1. View the introduction to the Real-Time Data Center and scroll down to the bottom of the page, click Subscribe Storage.

  2. Choose the provider for MongoDB Atlas services, deployment region, specification, and subscription period as follows: Purchase MongoDB Atlas and Storage

    • Cloud Provider: Currently supported: Google Cloud.

    • Region: Select the deployment region. Choose a region close to your data source for minimal network latency.

    • Specification: Pick the specification and storage size for MongoDB Atlas.

      tip

      TapData offers a free trial option with specifications that you can select. You can choose the Free Trial option to get started.

      Specifications Description
      • M10: 2 vCPUs, 2 GB RAM
      • M20: 2 vCPUs, 4 GB RAM
      • M30: 2 vCPUs, 8 GB RAM
      • M40: 4 vCPUs, 16 GB RAM
      • M50: 8 vCPUs, 32 GB RAM
      • M60: 16 vCPUs, 64 GB RAM

    • Subscription Period: Select the desired subscription period.

      Want to use an existing MongoDB Atlas?
      At the top of the page, click on click here to privede the connection information, and fill in the MongoDB Atlas connection URL.

  3. Click Subscription, on the following page, carefully review and confirm the specifications you wish to purchase. Ensure that the selected billing method aligns with your preferences. Additionally, verify that the email address provided is accurate and where you would like to receive the bill.

  4. Once you have double-checked all the information, click on the Pay Now button to proceed with the purchase.

  5. You will redirected to payment page. Please follow the instructions on the payment page to complete the payment process.

    After the payment is completed, the page will return to the Real-Time Data Hub page. Once the instance is automatically deployed, the page will be organized and displayed according to the hierarchy we introduced before. For information on how to use it, see Real-Time Data Hub Dashboard Introduction.