Configure a pipeline
When you create a pipeline, configure the following settings:Select destination
Choose the destination for your data. You have to configure at least one destination before you can create a pipeline.Handle schema changes
How Orka handles changes
When schemas change in the source database, intelligently:- Keep unaffected data flowing - other fields continue working normally
- Track changes - metadata records alterations
- Apply transformations as needed:
- Dropped columns: safe defaults or nulls
- Renamed columns: automatic mapping
- New columns: appropriate handling
Impact on your pipeline
Orka isolates changes to affected columns. Other columns continue working reliably.Pipeline lifecycle
Create a pipeline
- Discover data in the data catalog
- Select tables and columns needed
- Configure destination and settings
- Orka provisions infrastructure
Monitor a pipeline
Orka provides visibility into:- Data flow rates and volumes
- Schema change events
- Error rates and types
- Pipeline health status
Pipeline statuses
Orka tracks pipeline health through status indicators:| Status | Description |
|---|---|
| RUNNING | Pipeline is active and processing data normally |
| STARTING | Pipeline is initializing and preparing to process data |
| PAUSED | Pipeline is manually paused and does not process data |
| PARTIALLY_PAUSED | Some components of the pipeline are paused while others continue running |
| FAILED | Pipeline has encountered an error and stopped processing |
| UNKNOWN | Pipeline status can’t be determined (usually a temporary state) |
Check the pipeline details page for specific error messages and recommended actions when a pipeline shows FAILED status.
Maintain a pipeline
Check pipeline status regularly and address errors when they occur.Delete a pipeline
Always check for active pipelines before you delete connections.Our recommendations
- Start with non-sensitive data - create your first few pipelines with non-sensitive data to understand the workflow
- Monitor schema changes - monitor schema change notifications to understand how your source systems evolve
- Test in development first - test pipelines in development environments before deploying to production
Troubleshoot
Why did my pipeline break after I deleted a connection?
Why did my pipeline break after I deleted a connection?
If you delete a connection that’s being used in an active pipeline, the pipeline will break.To fix:
- Recreate the deleted connection with the same configuration, or
- Create a new pipeline with a different connection
Why is my pipeline not processing data?
Why is my pipeline not processing data?
Check these common causes:Source connection issues
- Test the source connection
- Verify database credentials are still valid
- Check network connectivity
- Test the destination connection
- Verify write permissions
- Check storage capacity
Why is creating a new data protection rule impacting my active pipelines?
Why is creating a new data protection rule impacting my active pipelines?
Creating new data protection rules after you publish tables triggers a re-scan of all tables, which can temporarily affect active sync pipelines.
Create and test all necessary data protection rules before you publish tables to the catalog.