What the Databricks REST API can and can't automate in a large migration

Every large Databricks migration — Hive metastore to Unity Catalog, on-prem Spark to Databricks, or consolidating a sprawl of workspaces into a governed few — starts with the same hopeful question: “can we just script this?” The honest answer is that the Databricks REST API automates the mechanical 80% of a migration extremely well, and the remaining 20% is exactly the part a script can’t safely decide for you. Knowing which is which up front is what separates a migration that finishes on schedule from one that stalls in month four.

What the API automates well

Workspace object migration. The Workspace API exports and imports notebooks, folders, and Repos in bulk — this is genuinely close to a solved problem, and it’s usually the first thing a migration script gets working, because it’s low-risk and easy to verify (diff the exported source against the imported copy).

Jobs and cluster policies. The Jobs API and Clusters API let you enumerate every job definition, schedule, and cluster policy in a source workspace and recreate them programmatically in a target one. Recreating a thousand job definitions by hand in the UI isn’t a realistic option; scripting it against the API is.

Identity and access provisioning. SCIM support lets you migrate users, groups, and group memberships without manually recreating every account. Combined with the Permissions API, you can replicate who has access to which jobs, clusters, and notebooks — as data, at least.

Unity Catalog object creation. Catalogs, schemas, external locations, and storage credentials can all be created via API from a defined inventory, which matters because a large migration usually means creating dozens or hundreds of these objects consistently rather than one-off through the UI.

What it can’t automate — because the hard part isn’t the API call

Hive ACLs don’t map to UC grants mechanically. Hive metastore’s table access control is enforced by cluster configuration — table ACL cluster settings, instance profiles scoped to S3 prefixes — not by grants stored with the table. There’s no API call that reads “who could access this table under the old model” and emits the equivalent GRANT statements, because the old model didn’t store that information in a form a script can reliably reconstruct. Someone has to decide the new grant structure; the API only applies the decision once it’s made.

DBFS mount paths are a business-logic problem wearing a technical costume. Every notebook, job, and library hardcoded to /mnt/some-path needs to be found and updated to a Unity Catalog volume or external location. The API can help you find every reference (grep the exported notebook source), but rewriting each one correctly requires understanding what that notebook is actually doing — a script that blindly find-and-replaces mount paths will break notebooks that reference paths conditionally or construct them dynamically.

Rate limits mean “migrate everything at once” isn’t a real strategy. The REST API enforces per-workspace rate limits on list and get calls, and a workspace with ten thousand notebooks or a thousand job definitions will throttle a naive single-loop migration script. A migration tool that actually works at scale batches requests and backs off on 429s — this is ordinary API-client engineering, but it’s exactly the kind of detail that turns a weekend script into something that silently fails halfway through a Friday-night migration window.

Correctness after translation isn’t verifiable by the API. Once ACLs are translated to grants and mount paths are rewritten to volumes, the API can tell you the objects were created — it cannot tell you the new permission model actually matches the intended one, or that a rewritten notebook still produces the same output. That validation is either manual spot-checking or a purpose-built comparison job, not a REST endpoint.

The shape of a migration that actually works

The migrations that go smoothly split into three phases, and the API’s role is different in each: an inventory phase that’s pure API reads — enumerate everything, understand scope, no changes made — a translation phase that’s mostly human judgment producing a reviewed mapping (old ACL → new grant, old mount path → new volume), and an apply phase where the API executes that already-reviewed mapping in controlled, rate-limited batches. Skipping straight from inventory to apply, treating the translation step as something the script can infer, is where migrations that looked automatable on paper turn into months of cleanup.

Zephico is a Databricks Consulting Partner, and our Databricks-certified engineers run large workspace and Unity Catalog migrations for clients — including the judgment calls in the translation phase that no API call makes for you. If you’re scoping a migration and want an honest read on what’s actually scriptable versus what needs a decision first, talk to us.