Achieving Near Real-Time Analytics with Secure, Hybrid Data Movement | Mobisy
Executive Summary: Mobisy Data Latency Reduction
Mobisy, a leading retail analytics platform, was hampered by a rigid data infrastructure that caused a 6-hour reporting delay and failed to meet strict internal data privacy compliance standards.
To address this dual challenge, EnHub.ai engineered a secure, cross-cloud solution utilizing the Fivetran Hybrid Deployment model. This architecture deployed the Fivetran agent directly within Mobisy’s private GCP Virtual Private Cloud (VPC), establishing a dedicated, encrypted data pipeline from their AWS RDS source to Google BigQuery.
The Client
Mobisy is a leading retail analytics platform providing critical insights into retail operations and sales performance for its global customers. Their reliance on accurate, timely data is fundamental to their service offering and customer value proposition.
The Business Challenge
Moboby’s existing data architecture utilized AWS RDS (MySQL) as its primary transactional database, with Amazon Redshift serving as its analytics data warehouse. This setup presented two critical hurdles:
A. Significant Data Latency: The existing ETL process was inefficient, resulting in a 6-hour delay in data synchronization. This meant that operational reports and critical business dashboards were consistently out of date, preventing the business from achieving near real-time visibility into key performance metrics.
B. Strict Data Privacy and Compliance: Mobisy operates under rigorous data governance policies. The customer explicitly required a solution that did not involve any external third-party SaaS tools processing or storing raw data outside of their own cloud infrastructure (VPC boundary). This eliminated most conventional cloud-based ETL solutions.
The EnHub.ai Solution Approach
EnHub.ai designed and implemented a secure, controlled, and low-latency solution leveraging the Fivetran Hybrid Deployment model. This architecture allowed Fivetran's powerful data extraction and loading capabilities to be executed entirely within Mobisy’s controlled environment, ensuring data integrity and compliance.
Key Implementation Details:
VPC-Internal Agent Deployment: The Fivetran Hybrid Agent was deployed onto a Google Kubernetes Engine (GKE) cluster, running entirely within Mobisy’s private GCP Virtual Private Cloud (VPC).
Secure Cross-Cloud Connectivity: To connect the source and destination databases across cloud providers, a secure VPN tunnel was established between the private AWS RDS MySQL instance and the GKE-based Fivetran agent residing in the GCP VPC.
Data Warehouse Migration: Amazon Redshift was replaced with Google BigQuery as the analytics destination, aligning the architecture with Mobisy's strategic future cloud direction.
Optimization: Sync intervals and parallel extraction settings were meticulously tuned to maximize throughput and minimize data lag between the transactional source and the data warehouse.
This unique architecture ensured that all data movement, processing, and encryption occurred within the customer’s controlled network, satisfying all compliance requirements.
Key Results and Achievements
The implementation of the Fivetran Hybrid Deployment successfully transformed Mobisy’s data synchronization process.
Metric | Before EnHub.ai Solution | After EnHub.ai Solution | Improvement |
|---|---|---|---|
Data Latency | 6 hours | < 30 minutes | Over 90% Reduction |
Data Compliance | Constraint | 100% Achieved | Full Adherence |
ETL Management | Manual Processes | Fully Managed (Fivetran) | Simplified Operations |
Challenges & Resolutions
Challenge | Resolution Implemented |
|---|---|
Private RDS Not Publicly Accessible | Established secure, encrypted VPN connectivity between the AWS and GCP VPCs, enabling private communication. |
Initial Data Transfer Bottlenecks | Finely tuned the Fivetran sync frequency and horizontally scaled the GKE node performance to accommodate higher data volumes. |
Compliance Restrictions on External Data Access | Utilized the Fivetran Hybrid model, where the agent processes and encrypts all raw data inside the customer’s VPC, ensuring no sensitive data ever leaves the controlled network. |
Business Impact
This data infrastructure overhaul has positioned Mobisy as a true near real-time analytics platform, delivering profound benefits to the business:
Improved Decision-Making: Faster data availability enables operations and sales teams to act on fresher insights, leading to more accurate forecasting and tactical decisions.
Enhanced Customer Value: The ability to provide near real-time dashboards and reports strengthens Mobisy’s core service offering.
Strengthened Governance: The project achieved enhanced confidence in data privacy and governance, eliminating previous security concerns about external data exposure.
Future-Proofing: The solution now serves as a reference architecture for Mobisy's expansion, demonstrating a model for secure, low-latency, and cross-cloud data movement for their own customers.
Conclusion
The successful implementation for Mobisy showcases EnHub.ai's capability to navigate the most complex data architecture challenges, specifically those involving cross-cloud integration and stringent compliance requirements. By leveraging the Fivetran Hybrid model, we delivered a strategic platform that transformed a significant operational bottleneck into a competitive advantage. This project stands as proof that speed and data privacy are not mutually exclusive, establishing a new and secure standard for near real-time data movement that empowers Mobisy to confidently scale its analytics offerings and secure its future growth
Social Media