Enterprise Databricks migration for a Canadian financial services company

Migrated 15,000 Airflow and Azure Data Factory jobs to UC-native Databricks Workflows in four months, with Terraform CI/CD and REST API automation doing the heavy lifting instead of a large migration team.

Client

Canadian financial services company (anonymized)

Industry

Financial Services

Region

Canada

Duration

4 months

The problem

The client was running roughly 15,000 orchestration jobs across Airflow and Azure Data Factory, feeding legacy Databricks workloads still on the Hive metastore — no unified governance, no UC-native permissions, and a job count large enough that a manual, one-by-one migration would have taken years.

What we built

A Unity Catalog migration paired with a full job-orchestration rebuild: all ~15,000 Airflow and ADF jobs converted to UC-compatible Databricks Jobs and Workflows, with the Databricks REST API driving the bulk of the conversion instead of hand-migrating each job, and Terraform managing deployment and version control for the result. As part of the engagement we also designed the client's SDLC end to end, as an org-wide best practice rather than a one-off fix for this migration.

Zephico ran this as a project delivery, not a staff-augmentation engagement — the client needed the migration done and governed correctly, not a larger team embedded long-term.

The scale problem

15,000 jobs is not a number you migrate by hand. Converting each one individually — remapping paths, rewriting cluster access into UC grants, testing, redeploying — would have taken years at any reasonable pace. The only way to hit a four-month timeline was to automate the conversion itself, using the Databricks REST API to drive job creation and configuration at scale rather than working through a spreadsheet of jobs one at a time.

What shipped

All ~15,000 Airflow and Azure Data Factory jobs converted to UC-compatible Databricks Jobs and Workflows, tested, and deployed to production. Terraform manages the deployment pipeline going forward, so job changes are version-controlled and reviewable instead of being clicked through a UI. Unity Catalog governance replaced the legacy Hive metastore setup across the estate.

Beyond the migration itself, we designed the client’s software development lifecycle end to end — not specific to this project, but as the process the org runs on for engineering work generally.

The team

Two data engineers and one solutions architect delivered the entire migration, from planning through production go-live, in four months. That’s a deliberately lean team for the scope — the kind of migration a larger systems integrator would typically staff much heavier, and price accordingly.

If you’re looking at a similar-scale migration and wondering whether it needs a large team to execute, that’s exactly the kind of assessment we do as part of Databricks data engineering work. Zephico is a Databricks Consulting Partner, and this engagement is packaged as a fixed-scope offer. Get in touch if you want us to look at your setup.