Monorepo for Databricks Zerobus Ingest SDKs.

GA: This SDK is generally available and supported for production use cases. Minor and patch version updates will not contain breaking changes. Major version updates may include breaking changes.

We are keen to hear feedback from you. Please file issues, and we will address them.

Zerobus is a high-throughput streaming service for direct data ingestion into Databricks Delta tables, optimized for real-time data pipelines and high-volume workloads.

We try to provide prebuilt native binaries for the following platforms:

Note: We do not currently have macOS CI runners, so macOS binaries are built locally and may not be available for every SDK or release. If your platform is not supported or you encounter compatibility issues, you can build from source or file an issue.

Before using any SDK, you need the following:

After logging into your Databricks workspace, look at the browser URL:

https://<databricks-instance>.cloud.databricks.com/o=<workspace-id>

• Workspace URL: The part before /o= (e.g., https://dbc-a1b2c3d4-e5f6.cloud.databricks.com)
• Workspace ID: The part after /o= (e.g., 1234567890123456)

Note: The examples above show AWS endpoints (.cloud.databricks.com). For Azure deployments, the workspace URL will be https://<databricks-instance>.azuredatabricks.net.

Create a table using Databricks SQL:

CREATE TABLE <catalog_name>.default.<table_name> (
    device_name STRING,
    temp INT,
    humidity BIGINT
)
USING DELTA;

Replace <catalog_name> with your catalog name (e.g., main).

1. Navigate to Settings > Identity and Access in your Databricks workspace
2. Click Service principals and create a new service principal
3. Generate a new secret for the service principal and save it securely
4. Grant the following permissions:
   • USE_CATALOG on the catalog (e.g., main)
   • USE_SCHEMA on the schema (e.g., default)
   • MODIFY and SELECT on the table

Grant permissions using SQL:

-- Grant catalog permission
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service-principal-application-id>`;

-- Grant schema permission
GRANT USE SCHEMA ON SCHEMA <catalog_name>.default TO `<service-principal-application-id>`;

-- Grant table permissions
GRANT SELECT, MODIFY ON TABLE <catalog_name>.default.<table_name> TO `<service-principal-application-id>`;

The service principal's Application ID is your OAuth Client ID, and the generated secret is your Client Secret.

Pick the API that matches your data.

Supported by all SDKs. Records are sent over a gRPC stream in one of two serialization formats:

• JSON - Simple, schema-free ingestion. Pass a JSON string or native object (dict, map, etc.) and the SDK serializes it. No compilation step required. Good for getting started or dynamic schemas.
• Protocol Buffers - Strongly-typed, schema-validated ingestion. More efficient over the wire. Recommended for production workloads.

Use proto2 syntax with optional fields to correctly represent nullable Delta table columns.

Instead of writing .proto files by hand, each SDK ships a tool to generate protobuf schemas directly from an existing Unity Catalog table. See the individual SDK READMEs for language-specific usage.

Available in the Rust, Python, Go, TypeScript, and Java SDKs starting from version 2.0.0, and in the C++ SDK from its initial 0.1.0 release. Currently in Beta — the API is stabilising but may still change before reaching GA. A third record format option alongside JSON and Protocol Buffers: send Apache Arrow RecordBatch data directly to Zerobus over the Arrow Flight protocol, on the same gRPC connection. Best fit when:

• Your workload is naturally columnar or batched — analytics pipelines, gateways aggregating short windows of rows, wide/numeric schemas where row-by-row serialization adds noticeable CPU overhead.
• Your application already produces Arrow data — pyarrow, the arrow-rs crates, DataFusion, Polars, or other libraries built on Arrow.

For sparse, one-row-at-a-time traffic, JSON or Protocol Buffers over the standard SDK gRPC path are usually simpler. Most SDKs ship a runnable examples/arrow/ directory; the C++ SDK covers Arrow Flight in its README until its examples land.

Ingestion is asynchronous in every SDK. An ingest call returns as soon as the record is queued — the SDK sends it and tracks its acknowledgment on a background task. To confirm that records were durably committed, call flush(); it returns once everything queued so far has been acknowledged.

The idiomatic flow is therefore ingest in a loop, then flush() — once at the end of a bounded batch, or periodically for a long-running stream. Where the SDK supports it, you can instead register an ack callback and be notified as records commit, without blocking at all.

Each ingest also returns the record's offset, and wait_for_offset(offset) blocks until that offset is acknowledged. That's useful when a particular record must be confirmed before you continue; because acknowledgments are ordered, waiting on the last offset of a run confirms the whole run. The one thing to avoid is waiting on every record inside a tight loop — that turns the asynchronous pipeline into a synchronous request/response and limits throughput to a single record per network round-trip.

See each SDK's README for exact method names and a runnable example.

All SDKs support HTTP CONNECT proxies via environment variables, following gRPC core conventions. The first variable found (in order) is used:

The no_proxy value is a comma-separated list of hostnames (suffix-matched) or * to bypass the proxy entirely.

export https_proxy=http://my-proxy:8080
export no_proxy=localhost,127.0.0.1

The SDK establishes a plaintext HTTP CONNECT tunnel through the proxy, then performs a TLS handshake end-to-end with the Databricks server. The proxy never sees decrypted traffic.

See CONTRIBUTING.md. Each SDK also has its own contributing guide with language-specific setup instructions.

This project is licensed under the Apache License 2.0. See LICENSE for the full text.