Configure Databricks - Orka Documentation

Before you connect Orka to Databricks, you need four things from your Databricks workspace.

Prerequisites

You need:

Access to a Databricks workspace with Unity Catalog enabled
A catalog and schema where Orka can write tables (or permission to create them)
Permission to create access tokens
Permission to create tables in your target schema

If you don’t have access to Databricks or Unity Catalog, contact your Databricks administrator.

Find your workspace URL

Your workspace URL is the web address you see when you log into Databricks.

Log into your Databricks workspace in a browser.
Look at the URL bar at the top of your browser.
Copy the URL up to and including .com or .net.

Workspace URL format

The format varies by cloud provider: AWS Databricks:

https://your-workspace-name.cloud.databricks.com

Azure Databricks:

https://adb-1234567890123456.19.azuredatabricks.net

GCP Databricks:

https://your-workspace-name.cloud.databricks.com

Don’t include any path after .com or .net. Use only the base workspace URL.

Create an access token

An access token lets Orka authenticate to your Databricks workspace. You can use either a personal access token (for testing) or a service principal token (for production).

Create a personal access token

Use personal access tokens for development and testing.

In your Databricks workspace, click your username in the top-right corner.
Select Settings from the dropdown menu.
Click Developer in the left sidebar.
Next to “Access tokens”, click Manage.
Click Generate new token.
Enter a description (for example, “Orka production connection”).
Set the token lifetime in days (90 days or less recommended).
Click Generate.
Copy the token immediately (a long alphanumeric string).
Click Done.

If you close the dialog, you won’t be able to view the token again and will have to generate a new one.

Token security

Databricks automatically revokes tokens that have not been used for 90 days. Treat tokens like passwords and store them securely.

Use service principals for production

For production, use service principals instead of personal access tokens. Service principals ensure pipelines continue working when team members leave.

Ask your Databricks administrator to create a service principal.
Ask them to generate an OAuth token or personal access token for the service principal.
Use that token in Orka.

Service principal setup requires Databricks administrator privileges. See Databricks service principal documentation for setup instructions.

Find or create a catalog

A catalog is a top-level container in Unity Catalog that organizes your data. The catalog has to exist before you connect Orka.

Find existing catalogs

Using Catalog Explorer (UI):

Log into Databricks.
Click Catalog in the left sidebar.
You see a list of available catalogs.
Note the catalog name you want to use.

Using SQL:

SHOW CATALOGS;

Create a catalog

Using SQL:

CREATE CATALOG IF NOT EXISTS orka_data
COMMENT 'Catalog for Orka data pipelines';

Using Catalog Explorer (UI):

Click Catalog in the left sidebar.
Click Create catalog.
Enter a catalog name (for example, orka_data).
Optionally add a description.
Click Create.

You need metastore admin privileges or CREATE CATALOG permission to create catalogs. Contact your Databricks administrator if you cannot create catalogs.

Typical catalog names

Organizations often create dedicated catalogs for different purposes:

prod_analytics - production analytics data
orka_data - dedicated for Orka pipelines
lakehouse - general lakehouse data
raw_data - raw operational data

Find or create a schema

A schema is a container within a catalog that holds tables. The schema has to exist before you connect Orka.

Find existing schemas

Using Catalog Explorer (UI):

Click Catalog in the left sidebar.
Expand your catalog name.
You see schemas listed underneath (for example, default, raw, staging).
Note the schema name you want to use.

Using SQL:

SHOW SCHEMAS IN your_catalog_name;

Create a schema

Using SQL:

CREATE SCHEMA IF NOT EXISTS my_catalog.orka_tables
COMMENT 'Schema for Orka pipeline tables';

Using Catalog Explorer (UI):

Click Catalog in the left sidebar.
Click on your catalog name.
Click Create schema.
Enter a schema name (for example, orka_tables).
Optionally add a description.
Click Create.

Common schema names

default - automatically created with every catalog
raw - raw data from sources
staging - intermediate data
orka_tables - dedicated for Orka

Orka creates one table in this schema for each source table you publish. For example, if you publish a customers table, Orka creates my_catalog.orka_tables.customers in Databricks.

Grant permissions

The user or service principal you use needs specific permissions on the catalog and schema.

Required permissions

-- Grant catalog access
GRANT USE CATALOG ON CATALOG my_catalog TO `[email protected]`;

-- Grant schema access
GRANT USE SCHEMA ON SCHEMA my_catalog.my_schema TO `[email protected]`;

-- Grant table creation
GRANT CREATE TABLE ON SCHEMA my_catalog.my_schema TO `[email protected]`;

-- Grant table modification
GRANT MODIFY ON SCHEMA my_catalog.my_schema TO `[email protected]`;

-- Grant table read access
GRANT SELECT ON SCHEMA my_catalog.my_schema TO `[email protected]`;

Who can grant permissions

These users can grant permissions:

Databricks metastore admin
Catalog owner
Schema owner
Users with GRANT privilege on the catalog or schema

Contact your Databricks administrator if you cannot grant permissions yourself.

Permission inheritance

Unity Catalog uses permission inheritance:

Permissions granted on a catalog apply to all schemas and tables in it
Permissions granted on a schema apply to all tables in it

Grant Orka permissions at the schema level (not catalog level) to limit access to only the schema Orka needs.

Verify your setup

Pre-connection checklist

You have your workspace URL (copied from browser address bar)
You have an access token (a long alphanumeric string)
Your catalog exists (verified in Catalog Explorer)
Your schema exists (verified in Catalog Explorer)
You have the required permissions

Test your permissions

Run these commands in a Databricks notebook:

-- Test catalog access
USE CATALOG my_catalog;

-- Test schema access
USE SCHEMA my_schema;

-- Test table creation
CREATE TABLE IF NOT EXISTS test_orka_connection (id INT);
DROP TABLE test_orka_connection;

Verify permissions

-- Check catalog permissions
SHOW GRANTS ON CATALOG my_catalog;

-- Check schema permissions
SHOW GRANTS ON SCHEMA my_catalog.my_schema;

Connect from Orka

Go to Destinations in Orka.
Choose Databricks as the destination type.
Enter the values you gathered:
- workspace URL
- access token
- catalog name
- schema name
Click Test connection.
Click Continue to save.

See Configure destinations.

Troubleshoot

Cannot find workspace URL

The workspace URL is in your browser’s address bar when you log into Databricks. Copy only the base URL (up to .com or .net), not the full path.If you have multiple workspaces, use the URL for the workspace where your catalog and schema exist.

Token generation button is disabled

You do not have permission to create tokens or your organization has disabled personal access tokens.Contact your Databricks administrator. They can grant you token creation permissions or create a service principal token for you.

Cannot see any catalogs

Unity Catalog is not enabled in your workspace or you do not have access to any catalogs.

Verify Unity Catalog is enabled (ask your Databricks administrator).
Ask your administrator to grant you USE CATALOG permission on at least one catalog.
Or ask them to create a catalog for Orka.

Cannot create catalog or schema

You do not have CREATE CATALOG or CREATE SCHEMA permissions.Contact your Databricks administrator. They can:

Grant you the necessary permissions.
Create the catalog and schema for you.
Provide access to an existing catalog and schema.

Test connection fails with permission error

Error	Cause	Solution
Catalog not found	Catalog does not exist or you lack `USE CATALOG`	Verify catalog exists: `SHOW CATALOGS`
Schema not found	Schema does not exist or you lack `USE SCHEMA`	Verify schema exists: `SHOW SCHEMAS IN catalog`
Cannot create table	Missing `CREATE TABLE` permission	Run: `SHOW GRANTS ON SCHEMA catalog.schema`
Cannot modify table	Missing `MODIFY` permission	Ask admin to grant `MODIFY` on schema

Run the permission grant SQL commands in the Grant permissions section or ask your Databricks administrator to grant them for you.

Token authentication fails

Token is expired (90 days of inactivity), incorrect, incomplete, revoked or wrong format.

Generate a new token following the steps in Create an access token.
Copy the entire token (a long alphanumeric string).
Verify the token works by testing it with Databricks CLI or API.

​Prerequisites

​Find your workspace URL

​Workspace URL format

​Create an access token

​Create a personal access token

​Token security

​Use service principals for production

​Find or create a catalog

​Find existing catalogs

​Create a catalog

​Typical catalog names

​Find or create a schema

​Find existing schemas

​Create a schema

​Common schema names

​Grant permissions

​Required permissions

​Who can grant permissions

​Permission inheritance

​Verify your setup

​Pre-connection checklist

​Test your permissions

​Verify permissions

​Connect from Orka

​Troubleshoot

​Related resources

Prerequisites

Find your workspace URL

Workspace URL format

Create an access token

Create a personal access token

Token security

Use service principals for production

Find or create a catalog

Find existing catalogs

Create a catalog

Typical catalog names

Find or create a schema

Find existing schemas

Create a schema

Common schema names

Grant permissions

Required permissions

Who can grant permissions

Permission inheritance

Verify your setup

Pre-connection checklist

Test your permissions

Verify permissions

Connect from Orka

Troubleshoot

Related resources