Data Engineering

Fresh data
without hammering production

Reliable data pipelines are what reporting, analytics and every downstream system depend on. We design and build change data capture pipelines that keep your warehouse minutes behind the source, without analytical queries ever touching the database your business runs on.

Discuss a data engagement

What we do

Pipeline engineering

Log-based capture from Oracle, SQL Server, PostgreSQL, DB2 and MySQL, landed and merged into your warehouse continuously. HVR where its licence earns itself, custom tooling built on your database native change mechanisms where it does not.

Source system relief

Reporting queries move to a replica that exists to be queried. Log-based capture reads the transaction log rather than the tables, so the source carries the write path and nothing else. Your application team stops reviewing every new dashboard request.

ETL and ELT development

Ingestion and transformation on Glue, Data Factory, Dataflow, Snowflake and Databricks, with or without CDC underneath. Incremental by default, idempotent steps, and orchestration with real dependency handling rather than chained schedules.

Data quality and reconciliation

Row counts, checksums and business-key comparisons between source and target, running continuously rather than once at go-live. If the two ever disagree, you find out before your users do.

Monitoring and alerting

Replication lag, failed steps and schema changes surface as actionable alerts. A pipeline that fails silently and leaves stale numbers in a dashboard is the failure mode we see most often, so we build against it.

Documentation and handover

Every engagement ends with written runbooks and a walkthrough with the engineers who own the pipelines afterwards. The goal is that you do not need us for the next one.

From raw change to reliable reporting

Almost every estate starts with query-based extraction. A scheduled job selects from the source, writes to the target, and it works well while the tables are small.

Two things then happen. The extraction gets heavier as the data grows, so it competes with the application for exactly the resources the application needs most at exactly the busiest time. And the schedule stretches, because a full reload is the only thing that reliably produces a correct target, and it slows down every quarter. Teams respond by running it less often, which makes the data staler, which is the opposite of what everyone wanted.

Change data capture removes that trade-off. Rather than repeatedly asking the source what it holds now, you read the transaction log the database already writes and apply only what changed. Load on the source drops and stops scaling with data volume, freshness improves from hours to minutes, and the full reload becomes a recovery tool rather than the daily routine.

We have delivered this work in regulated financial services, where the reporting estate could not be allowed to affect the production write path and every applied change had to be accounted for afterwards.

Change data capture pipelines

The engineering is mostly in the details that only surface under load.

We start with what your sources actually support. Log-based capture depends on the database being configured for it, supplemental logging on Oracle, an appropriate replication setting on PostgreSQL, and that answer determines what is possible before any tool is selected. From there: which tables genuinely need replicating rather than all of them, how changes are landed and merged into the target, what happens to deletes, and how far behind the target is allowed to drift before someone is alerted.

The properties we build for are restartability from a known position after any interruption, idempotent application so a replay cannot double-count, deliberate handling of schema drift, and reconciliation that runs continuously rather than once at go-live.

On HVR specifically. Where it earns its cost is heterogeneous capture at volume, several different source database platforms under one operational model, with mature handling of the awkward parts. It is also frequently sold into estates that would be served just as well by a native connector at a fraction of the price. We will tell you which one you are, and we hold no commercial interest in the answer.

Where a licence is not justified, custom tooling built on the change mechanisms your database already provides does the same job, and you own it afterwards.

ETL pipelines on cloud platforms

Not every pipeline needs CDC underneath it, and a good deal of this work is straightforward pipeline development.

We build ingestion and transformation on AWS (Glue, Redshift, S3), Azure (Data Factory, Synapse, Fabric), GCP (BigQuery, Dataflow), and into Snowflake and Databricks, as well as onto database servers you run yourself. The platform shapes the implementation, but the properties do not change: incremental rather than full reload, idempotent steps so a retry is safe, orchestration with real dependency handling rather than jobs chained by start time and hope, and failures that surface as actionable alerts.

That last point matters more than it sounds. A pipeline that fails loudly is an inconvenience. A pipeline that fails quietly and leaves stale numbers in a dashboard someone is making decisions from is a different category of problem, and it is the failure mode we see most often in estates that have grown without review.

Who this is for

A good fit if your reporting is constrained by a batch window that no longer fits, or your production databases carry analytical load they were never meant to, and you want the team that operates the pipelines afterwards to have been involved in building them.

Not the right fit if you are looking for someone to own your data platform indefinitely. We build, document and hand over. If what you need is a permanent data team, hiring one is the better answer and usually the cheaper one.

Documentation and handover

We write comprehensive technical documentation as part of delivery, because an engagement that leaves your team dependent on us has failed at the thing you were buying. Every engagement ends with a walkthrough session with the engineers who own the pipelines afterwards.

What it costs

Scoped after a short call. This work varies too much on source platform, change volume and the number of systems involved for a published price to be honest, and a short assessment is usually what determines whether the build is two weeks or two months.

Advisory and smaller scoped work runs at EUR 100 per hour. All prices exclude VAT.

Frequently asked

How current does the target stay?
Typically seconds to low minutes behind the source, depending on change volume and how far the data travels. The real constraint is rarely the capture itself, it is the merge into the target. A warehouse that dislikes frequent small writes sets a floor on how often you can apply changes. We size that against what your reporting genuinely needs, because a dashboard read once each morning does not justify the cost of second-level freshness.
What load does capture put on our production database?
Log-based capture reads the transaction log rather than querying the tables, so the overhead on the source is small and, more importantly, it does not compete with your application for the same locks and buffers. That is the entire reason to prefer it over query-based extraction, which gets heavier exactly when the database is already busy.
Do we need HVR, or is it overkill?
It earns its licence when you capture from several heterogeneous databases at high change volume and want one operational model across all of them. For a single PostgreSQL source, a modest change rate, or a target whose platform already ships a competent connector, custom tooling or the native service is usually the better answer. We establish which of the two applies before anyone signs a licence, and we hold no reseller relationship either way.
What happens when the source schema changes?
It is the most common way these pipelines break, so it is designed for rather than discovered. Additive changes propagate automatically. Destructive ones stop the pipeline deliberately instead of silently dropping a column downstream, because a pipeline that keeps running while losing data is worse than one that stops.
Which cloud platforms do you build on?
AWS, Azure and GCP, plus Snowflake and Databricks as targets, and database servers you run yourself. The platform shapes the implementation and the cost model, but the engineering properties we build for do not change.
Do you build the pipelines or just advise?
Either. Most engagements are build work with your engineers alongside, so the people who maintain the pipelines have seen them being written. Advisory reviews of an existing design are also fine, and are a considerably cheaper way to find out that something will not scale.
Can you work on pipelines that have no CDC in them at all?
Yes. A good deal of the work is ordinary batch ETL and ELT, files and APIs into a warehouse, transformation layers, orchestration. CDC is one ingestion pattern among several, and applying it where it is not needed adds cost and moving parts.

Related services

Ready to talk?

Independent data engineering consultants for change data capture pipelines with HVR and custom tooling, and ETL pipelines on AWS, Azure, GCP, Snowflake and Databricks.

Get in touch