> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oleander.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Redpanda

> Stream Redpanda topics into your oleander Iceberg catalog with a Redpanda Connect pipeline.

[**Redpanda**](https://www.redpanda.com/) is a Kafka-compatible streaming platform. Its [Redpanda Connect](https://docs.redpanda.com/redpanda-cloud/develop/connect/about/) pipelines include an [Iceberg output](https://docs.redpanda.com/redpanda-connect/components/outputs/iceberg/) that writes messages to Iceberg tables through a REST catalog. Point it at your oleander catalog to land topics in your lake, ready to query across compatible engines.

![Redpanda Connect pipeline reading the hello-world topic, mapping each message, and writing to the hello\_world Iceberg table](https://mintcdn.com/oleander/2KQrezy9xZGMRF1-/images/redpanda-pipeline.webp?fit=max\&auto=format\&n=2KQrezy9xZGMRF1-\&q=85\&s=a474ce0e31b2f5d7fb840e7248ec404d)

## What you need

| Value | Where to find it |
| - | - |
| **Organization ID** | [Settings → Organization](https://oleander.dev/app/settings) |
| **API key** | [Settings → API keys](https://oleander.dev/app/settings/api-keys) |

The organization ID is both the OAuth client ID and the catalog warehouse name. The API key is the OAuth client secret.

You also need a Redpanda Cloud cluster. Connect pipelines run on Serverless, Dedicated, and BYOC clusters, and the `iceberg` output is available from Redpanda Connect 4.80.0.

## Setup

1. In the oleander [lake](https://oleander.dev/app/lake), hover over the `oleander` catalog in the sidebar, click **+**, enter `redpanda` as the namespace name, and click **Create Namespace**. This is the namespace the pipeline writes into.
2. In Redpanda, create a user for the pipeline under **Security → Users**, with ACLs to read your topic and its consumer group.
3. Open the cluster's **Secrets Store** and add three secrets:

| Secret | Value |
| - | - |
| `REDPANDA_USER` | The pipeline user's name |
| `REDPANDA_PASSWORD` | The pipeline user's password |
| `OLEANDER_API_KEY` | Your oleander API key |

4. Open **Connect**, click **Create a pipeline**, switch to the **YAML** tab, and paste this config, replacing `<organization-id>`:

```yaml theme={null}
input:
  redpanda:
    seed_brokers:
      - ${REDPANDA_BROKERS}
    topics:
      - hello-world
    consumer_group: oleander-iceberg
    tls:
      enabled: true
    sasl:
      - mechanism: SCRAM-SHA-256
        username: ${secrets.REDPANDA_USER}
        password: ${secrets.REDPANDA_PASSWORD}

pipeline:
  processors:
    # hello-world messages are plain text; the iceberg output needs JSON objects
    - mapping: |
        root.message = content().string()
        root.topic = @kafka_topic

output:
  iceberg:
    catalog:
      url: https://iceberg.oleander.dev/catalog
      warehouse: <organization-id>
      auth:
        oauth2:
          server_uri: https://oleander.dev/api/v1/oidc/token
          client_id: <organization-id>
          client_secret: ${secrets.OLEANDER_API_KEY}
          scope: catalog
    namespace: redpanda
    table: hello_world
    storage:
      aws_s3:
        # required by the schema but unused; oleander vends the real location and credentials
        bucket: oleander
        region: us-east-2
    schema_evolution:
      enabled: true
    batching:
      period: 60s
```

5. Click **Save**, then **Start pipeline**.

The `iceberg` output fields that connect it to oleander:

| Field | Value |
| - | - |
| `catalog.url` | `https://iceberg.oleander.dev/catalog` |
| `catalog.warehouse` | Your organization ID |
| `catalog.auth.oauth2.server_uri` | `https://oleander.dev/api/v1/oidc/token` |
| `catalog.auth.oauth2.client_id` | Your organization ID |
| `catalog.auth.oauth2.client_secret` | Your API key, read from a secret |
| `catalog.auth.oauth2.scope` | `catalog` |
| `namespace` | The Iceberg namespace to write into, created in step 1 |
| `table` | The table to write into. It is created on the first write. |
| `storage.aws_s3.region` | `us-east-2` |
| `schema_evolution.enabled` | `true`, so the output creates the table and adds columns as new fields appear |

> [!NOTE]
>
> Redpanda Connect exchanges your organization ID and API key for a short-lived catalog token, and oleander vends the storage credentials for each table. The `storage.aws_s3.bucket` field is required by the output's schema, but oleander supplies the real table location, so any value works and there are no S3 keys to configure.

The `iceberg` output writes JSON objects, inferring a column from each field. The `mapping` processor above wraps the plain-text messages from Redpanda's quickstart `hello-world` topic into `{message, topic}` objects. If your topic already carries JSON, drop the `pipeline` section.

Once the pipeline starts, its logs show the output creating the table on the first write:

![Redpanda Connect pipeline logs showing the iceberg output becoming active and creating the redpanda.hello\_world table](https://mintcdn.com/oleander/2KQrezy9xZGMRF1-/images/redpanda-pipeline-logs.webp?fit=max\&auto=format\&n=2KQrezy9xZGMRF1-\&q=85\&s=d9c766b411ccd74fda158c49c0c34511)

## Querying the data

Query the table like any other table in your catalog:

```sql theme={null}
SELECT * FROM oleander.redpanda.hello_world;
```

![The oleander lake querying redpanda.hello\_world, returning five rows from the hello-world topic](https://mintcdn.com/oleander/2KQrezy9xZGMRF1-/images/redpanda-query.webp?fit=max\&auto=format\&n=2KQrezy9xZGMRF1-\&q=85\&s=4de87aab8982b0456319ca34c4ce18ab)

Every batch is an Iceberg commit, and `batching.period` sets how often one lands. A longer period means fewer, larger files. For long-running pipelines, schedule [Iceberg maintenance](https://docs.oleander.dev/platform/storage/tables.md#table-health-and-maintenance) to compact small files, expire old snapshots, and keep query times flat.

For keyed topics, the output can upsert with `row_operation` and `identifier_fields`. Prefer `merge_strategy: copy-on-write` there: the default `merge-on-read` writes equality deletes, which slow down reads until maintenance compacts them.

> [!NOTE]
>
> This guide uses Redpanda Connect, not Redpanda's built-in Iceberg Topics. Iceberg Topics only runs on BYOC and BYOVPC clusters and writes its data files to Redpanda's own bucket, so it can't use the storage your oleander catalog provides.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.