Migrating Legacy Data Pipelines to Event-Driven Architecture: A Phased Playbook

It's 3am. Your phone buzzes. The nightly Spark job that feeds your ML feature store has been stuck for four hours, and now the fraud detection model is serving predictions on yesterday's data. Customers are filing chargebacks. Your on-call engineer is staring at a 9,000-line SQL transformation nobody has touched since 2021, trying to figure out which upstream table changed its schema.

This scenario plays out across hundreds of engineering organizations every week. The fix isn't patching the batch job. It's replacing it. Migrating legacy batch ETL pipelines to event-driven architecture (using Apache Kafka, Apache Pulsar, or a managed equivalent) lets you process data in seconds instead of hours. But the migration itself is what terrifies teams, and rightfully so. Production can't stop. Nobody fully documents the existing transformations. And the last team that tried a "full rewrite" blew past their deadline by a year.

The answer is the strangler fig pattern: a phased approach where you wrap existing batch pipelines, run event-driven replacements in parallel, validate equivalence, and cut over one pipeline at a time. No production freezes. No big-bang cutovers. This playbook walks you through every phase, from picking your first pipeline to monitoring the metrics that tell you when it's safe to flip the switch.

Your Batch ETL Pipeline Is Already Broken (You Just Haven't Noticed Yet)

Batch ETL was designed for a world where 24-hour data latency was acceptable. Monthly financial reports, weekly dashboards, overnight warehouse loads. That world no longer exists for most teams. AI inference pipelines, fraud detection, real-time personalization, and operational analytics all demand data freshness measured in seconds or minutes, not hours.

The symptoms of a broken batch pipeline are subtle at first. Your data science team starts building workarounds, pulling directly from production databases because the warehouse is too stale. Your product team ships a "real-time" feature that actually queries a table last refreshed six hours ago. Your finance team runs a report on Monday morning using Friday's data because the weekend batch window failed silently.

The fear of migration is rational. I've worked with teams that had batch pipelines running transformations written by engineers who left the company three years ago. The SQL does something critical, but nobody can explain exactly what. Rewriting everything from scratch feels like rebuilding an airplane while it's in flight.

The strangler fig pattern eliminates this fear by design. Named after the tropical fig that grows around a host tree and gradually replaces it, this pattern lets you wrap an existing pipeline, run a new event-driven version alongside it, prove equivalence, and then retire the old one. You never touch the original until you're confident the replacement works.

Why Big-Bang Rewrites Fail (and What to Do Instead)

Teams that attempt full pipeline rewrites consistently underestimate the effort. In our experience working with enterprise data teams, rewrite projects regularly exceed their original timeline estimates by 2x or more, and some never finish at all. The reasons are predictable: undocumented business logic embedded in SQL, hidden dependencies between pipelines, and shifting requirements during the rewrite window.

The strangler fig pattern applied to data pipelines works in three steps. First, you wrap the existing batch job by intercepting its inputs and outputs. Second, you build a new event-driven consumer that processes the same source data. Third, you validate that both produce identical results, then cut over. Each pipeline migrates independently, so a failure in one doesn't block the others.

A concrete example: you have a nightly Spark job that reads from a PostgreSQL staging table, applies 40 SQL transformations, and writes results to a Redshift warehouse. Instead of rewriting all 40 transformations at once, you set up a Kafka Connect CDC connector on the PostgreSQL source. A new Kafka Streams application consumes change events and applies the same transformations (decomposed into smaller units). Both the old Spark output and the new Kafka Streams output write to comparison tables. A nightly reconciliation job flags any divergence.

The strangler fig approach wins for the majority of teams because it constrains blast radius. If your first event-driven pipeline has a bug, you still have the batch job running. You fix the bug, re-validate, and try again. No production impact.

Picking Your First Pipeline (Hint: It's Not the Most Important One)

The instinct is to start with your most painful, highest-value pipeline. Resist it. Your first migration is a learning exercise. You will make mistakes in schema design, consumer configuration, and monitoring setup. You want those mistakes to happen on a pipeline where the blast radius is small.

Selection Criteria

Score each candidate pipeline across five dimensions:

- Data volume: Medium is ideal. Too small and you won't learn anything about throughput. Too large and you'll hit scaling issues before you've solved the basics.

- Downstream consumers: Fewer is better for your first migration. A pipeline feeding two dashboards is safer than one feeding 14 microservices.

- Transformation complexity: Start with pipelines that have clear, testable transformations. Avoid the 10,000-line SQL monolith for round one.

- Business criticality: Choose something important enough that people care about it, but not so critical that a one-hour outage triggers an executive incident review.

- Owner enthusiasm: This matters more than you'd expect. A team that actively wants to modernize will invest the effort to make it work. A team that's been assigned migration duty will drag their feet.

Rate each criterion 1-5, multiply business criticality by -1 (lower is better for your first pick), and sum the scores. The pipeline with the highest composite score is your starting point.

The Inventory Step You Can't Skip

Before touching any pipeline, trace its full lineage. Document every source table, every transformation step, and every downstream consumer. Use your data catalog if you have one. If you don't, spend a week building a lightweight lineage map. Tools like dbt's lineage graph, Apache Atlas, or even a spreadsheet will work. Hidden dependencies are the number-one cause of migration failures, and you can't find them after you've already started.

The Dual-Write Bridge: Running Batch and Event-Driven in Parallel

The dual-write pattern is the mechanical core of the strangler fig migration. Your source system writes to both the legacy batch store and a Kafka (or Pulsar) topic simultaneously. Both the old batch pipeline and the new event-driven pipeline process the same data. You compare outputs until they match consistently.

Here's a Python example of a dual-write producer that publishes to Kafka while still inserting into the batch staging table, with checksum validation:

import hashlib
import json
from confluent_kafka import Producer
import psycopg2

def dual_write(record: dict, kafka_producer: Producer, db_conn):
    """Write to both Kafka and PostgreSQL staging table with checksum."""
    payload = json.dumps(record, sort_keys=True, default=str)
    checksum = hashlib.sha256(payload.encode()).hexdigest()

    # Write to Kafka topic with checksum header
    kafka_producer.produce(
        topic="orders.events.v1",
        key=str(record["order_id"]).encode(),
        value=payload.encode(),
        headers={"checksum": checksum.encode()},
    )

    # Write to legacy staging table (existing batch pipeline still reads this)
    cursor = db_conn.cursor()
    cursor.execute(
        """INSERT INTO staging.orders (order_id, payload, checksum, created_at)
           VALUES (%s, %s, %s, NOW())""",
        (record["order_id"], payload, checksum),
    )
    db_conn.commit()

    return checksum

The checksum is critical. Without it, you have no way to verify that the same record produced the same output in both pipelines. During the reconciliation phase, a nightly job compares batch output rows against event-driven output rows, matching on business key and checksum. Any divergence above your configured threshold (we recommend starting at 0.03%) triggers an alert.

The reconciliation loop runs nightly during the parallel phase. It queries both output stores, joins on business keys, compares checksums, and writes divergence reports. Teams frequently tell us they discover bugs in the old batch pipeline during this phase, because the event-driven version processes edge cases differently (and often more correctly).

Schema Registries: The Insurance Policy Nobody Wants to Buy

Schema evolution is the number-one killer of event-driven migrations. A producer changes a field from user_name to username, deploys on Tuesday, and 14 downstream consumers silently start writing nulls into their databases. Nobody notices until a product manager asks why the user profile page is blank for everyone who signed up this week.

A schema registry prevents this by enforcing compatibility rules on every schema change. Before a producer can publish events with a new schema version, the registry validates that the change is backward-compatible (or forward-compatible, depending on your policy). Breaking changes are rejected at deploy time, not discovered in production.

Here's an Avro schema example showing a backward-compatible evolution (adding a field with a default):

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "order_id", "type": "string"},
    {"name": "amount_cents", "type": "long"},
    {"name": "currency", "type": "string", "default": "USD"},
    {"name": "customer_segment", "type": "string", "default": "unknown"}
  ]
}

The customer_segment field was added in v2 with a default value. Consumers still running v1 of the schema can deserialize v2 events because Avro ignores unknown fields and fills in defaults for missing ones. If you tried to remove the amount_cents field instead, the registry would reject the change under BACKWARD compatibility mode.

Practical advice: enforce BACKWARD compatibility mode by default. This means new schemas can add fields (with defaults) but can't remove or rename existing fields. Only escalate to FULL compatibility when you control every consumer and can coordinate simultaneous upgrades, which in practice means almost never.

For registry selection, Confluent Schema Registry is the standard for Kafka-native deployments. AWS Glue Schema Registry integrates well if your infrastructure is already on AWS and you want to avoid managing another service. Apicurio is the open-source alternative with broader protocol support. All three support Avro and Protobuf; JSON Schema support varies.

Handling the Transformation Layer Without Rewriting Business Logic

The biggest objection I hear from data engineering teams: "Our batch jobs have 10,000 lines of SQL with business logic nobody understands." Fair. But you don't need to rewrite it all at once.

The strategy is decomposition. Take the monolithic SQL, break it into discrete transformation steps, and implement each step as a stream processor. A 200-line SQL query with five joins, three subqueries, and a window function becomes five or six Kafka Streams operations, each testable in isolation.

Here's a batch SQL join and its Kafka Streams equivalent:

-- Batch: join orders with customer segments, aggregate daily revenue
SELECT c.segment, SUM(o.amount_cents) as daily_revenue
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.created_at >= CURRENT_DATE - INTERVAL '1 day'
GROUP BY c.segment;

// Kafka Streams: KTable-KTable join with windowed aggregation
KTable<String, Order> orders = builder.table("orders.events.v1");
KTable<String, Customer> customers = builder.table("customers.v1");

KTable<String, Long> revenueBySegment = orders
    .join(customers,
          order -> order.getCustomerId(),   // foreign key extractor
          (order, customer) -> new SegmentedOrder(customer.getSegment(), order.getAmountCents()))
    .groupBy((key, segOrder) -> KeyValue.pair(segOrder.getSegment(), segOrder))
    .aggregate(
        () -> 0L,
        (segment, segOrder, agg) -> agg + segOrder.getAmountCents(),
        (segment, segOrder, agg) -> agg - segOrder.getAmountCents(),
        Materialized.as("revenue-by-segment-store"));

To test equivalence, run property-based tests that feed the same input dataset through both the batch SQL and the stream processor, then compare outputs. Tools like Hypothesis (Python) or jqwik (Java) can generate edge-case inputs automatically. If both produce identical results across 10,000 generated test cases, you have high confidence in parity.

Tracking transformation parity metrics over time is where an engineering intelligence dashboard adds real value. You can monitor match rates, flag regressions, and correlate parity drops with specific code changes across your migration portfolio.

Monitoring the Migration: Metrics That Tell You When to Cut Over

You don't cut over when the migration "feels done." You cut over when three quantitative gates are met:

1. Data completeness: The event-driven pipeline processes ≥99.97% of the records the batch pipeline processes, measured over a rolling 7-day window.

2. Latency improvement: The event-driven pipeline delivers results in measurably less time than the batch window. Don't assume this. Measure it.

3. Consumer health: Downstream consumers show zero increase in error rates when reading from the new output store.

Consumer lag is the single most important metric during migration. In Kafka, consumer lag measures how far behind a consumer group is from the latest offset in a partition. During the parallel-run phase, rising consumer lag means your streaming pipeline can't keep up with the event volume. Set up PagerDuty or Opsgenie alerts for consumer lag exceeding 5 minutes. If lag grows consistently, you have a throughput problem to solve before cutover.

Use this phase checklist to track progress:

Frequently Asked Questions

How long does a full migration take?

Each individual pipeline takes 3 to 8 weeks from instrumentation to full cutover. An organization with 20 batch pipelines should expect 6 to 12 months for the complete migration, assuming 2 to 3 pipelines migrating in parallel after the first one proves the pattern.

Can we use managed services instead of self-hosted Kafka?

Yes, and for most teams I'd recommend it. Amazon MSK, Confluent Cloud, and Azure Event Hubs (Kafka-compatible) reduce operational overhead significantly. The migration pattern is identical; only the infrastructure provisioning changes.

What if our batch pipeline has side effects like sending emails or triggering API calls?

Isolate side effects into their own processing step. During the parallel-run phase, only the batch pipeline should execute side effects. The event-driven pipeline logs intent to trigger the side effect but doesn't execute it until after full cutover. This prevents duplicate emails and API calls.

How do we handle backfills in an event-driven architecture?

Use a dedicated backfill topic with a separate consumer group. Replay historical data from your batch store into the backfill topic, and let the streaming pipeline process it at its own pace. Don't mix backfill traffic with live traffic on the same consumer group; the lag metrics become meaningless.

After the Migration: What Changes About How Your Team Works

The technical shift is only half the story. Event-driven architecture changes how your team operates day to day.

On-call patterns shift. Instead of debugging failed cron jobs at 3am (which you discover at 7am when someone checks), problems surface as consumer lag alerts within minutes. The mean time to detection drops from hours to seconds. The 3am scenario from the opening of this article becomes a consumer lag alert at 3:01am, investigated and resolved by 3:20am, with no downstream impact.

Ownership models change too. In the batch world, a central data engineering team typically owns all pipelines. In an event-driven world, each domain team owns their topics, schemas, and consumers. The data engineering team shifts from pipeline operator to platform provider, maintaining the Kafka cluster, schema registry, and monitoring infrastructure while domain teams build and operate their own consumers.

SlopBuster's automated code review can catch common anti-patterns during this transition, like consumers without proper error handling, missing dead-letter queues, or schema changes that bypass the registry.

Your Concrete Next Steps

1. This week: Pick one pipeline using the scoring criteria above. Score your top three candidates and choose the one with the lowest business criticality and highest owner enthusiasm.

2. Within 30 days: Instrument that pipeline for dual-write. Set up a Kafka topic, deploy a CDC connector or dual-write producer, and run your first reconciliation comparison.

3. Track one metric: Consumer lag on your new event-driven pipeline. This single number tells you more about migration health than any status report.

That 6-hour batch job failing at 3am? It doesn't have to be your reality. One pipeline at a time, with validation at every step, you replace fragile overnight cron jobs with event-driven pipelines that process data as it arrives. Not through a heroic rewrite, but through disciplined, incremental migration that never puts production at risk.

References

[1] Thoughtworks, "Strangler Fig Application," Technology Radar. https://www.thoughtworks.com/radar/techniques/strangler-fig-application

[2] Confluent, "From Batch to Real-Time: The Architecture of Modern Data Pipelines," 2024. https://www.confluent.io/blog/batch-to-real-time-data-pipelines/

[3] Apache Software Foundation, "Kafka Streams Documentation: KTable-KTable Join," 2024. https://kafka.apache.org/documentation/streams/

[4] Confluent, "Schema Registry Overview," 2024. https://docs.confluent.io/platform/current/schema-registry/index.html

[5] Martin Kleppmann, "Designing Data-Intensive Applications," O'Reilly Media, 2017. Chapter 11: Stream Processing.

[6] Google Cloud, "DORA State of DevOps Report," 2024. https://dora.dev/research/

[7] Netflix Technology Blog, "Evolution of the Netflix Data Pipeline," 2024. https://netflixtechblog.com/