Poor performance coupled with high costs are the major challenges for today's data warehousing solutions. Some systems have to process terabytes of data to update data warehouses with tens of terabytes of data. Typical solutions involve some combination of micro-batching and higher performance hardware. However, both are now seen as stop-gap solutions, having proven unable to scale with increasing data volumes and complexity.
About Continuous ETL
Extract, Load and Transform (ETL) is the term used to describe a common approach for populating data warehouses. Most data warehouses require data to be consolidated across multiple source systems, where data structures and data granularity may differ.
However, ETL on its own is insufficient. The high volume of data and meta-data that is generated and captured by today's web applications can grind a data warehouse to a stand still. Data warehouses require aggregation of the vast volumes of raw data in order to extract meaningful business intelligence while its still valued.
SQLstream uses Continuous ETL to Eliminate Bottlenecks
SQLstream enhances the data warehouse by adding true, real-time capabilities that both increase scalability and help to keep it current while eliminating latency and down time. SQLstream uses a combination of continuous ETL and real-time SQL queries to offload heavy data processing, reduce replication issues and deliver real-time operational intelligence.
SQLstream enables multiple sources of heterogeneous data to be aggregated, correlated and filtered in real-time - before the ‘Load’ operation to the data warehouse takes place. Change Data Capture adapters provide the real-time ‘Extract’ function, delivering a stream of relational data from the source systems. SQLstream processes the relational data streams, providing both the ‘Transform’ and ‘Aggregation’ functions in a single platform.
Have questions about your current project? Click here to ask a SQLstream expert.