> ## Documentation Index
> Fetch the complete documentation index at: https://docs.oleander.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark

> Initialize workspaces, upload artifacts, submit jobs, and register clusters from the CLI.

Run your Spark applications on oleander-managed infrastructure or on your own registered clusters. For a full overview of Spark on oleander, see the [Spark documentation](/platform/compute/spark).

## Oleander-managed Spark

### Initialize a PySpark workspace

Create a new PySpark job workspace:

```bash theme={null}
oleander spark init <dirname>
```

**Example:**

```bash theme={null}
oleander spark init my-job
```

You can also initialize the current directory:

```bash theme={null}
oleander spark init .
```

Pass flags to skip the interactive prompts:

```bash theme={null}
oleander spark init my-job \
  --entrypoint-name pipeline.py \
  --custom-package-name mylib
```

| Flag | Description |
| - | - |
| `--entrypoint-name <file>` | Name for the main entrypoint file (must end with `.py`) |
| `--custom-package-name <name>` | Name for the custom Python package directory |

The initialized workspace includes:

* `entrypoint.py` as the Spark job entrypoint
* `mylib/` for Python modules packaged as `pyFiles`
* `pyproject.toml` and `uv.lock` for project and dependency management with `uv`
* `Makefile` targets for building deployable artifacts

Use `uv` to manage dependencies:

```bash theme={null}
uv sync --dev
uv add <package>
uv add --dev <package>
```

Use `make` to build the deployment artifacts:

```bash theme={null}
make
```

This builds:

* `out/pyfiles.zip`
* `out/environment.tar.gz`

You can also build individual artifacts:

```bash theme={null}
make pyfiles
make environment
make rebuild
```

After building, upload and submit from the initialized workspace:

```bash theme={null}
oleander spark jobs upload entrypoint.py \
  --py-files out/pyfiles.zip \
  --virtualenv out/environment.tar.gz
```

```bash theme={null}
oleander spark jobs submit entrypoint.py \
  --namespace <namespace> \
  --name <job-name> \
  --wait
```

### List your Spark artifacts

```bash theme={null}
oleander spark jobs list
```

### Download a Spark artifact

Print the source of an uploaded artifact to stdout:

```bash theme={null}
oleander spark jobs get <entrypoint>
```

To fetch a specific version:

```bash theme={null}
oleander spark jobs get <entrypoint> --artifact-version 3
```

| Flag | Description |
| - | - |
| `--artifact-version <n>` | Version number to retrieve (defaults to latest) |

### Upload a Spark artifact

Upload a local `.py` or `.jar` artifact to oleander:

```bash theme={null}
oleander spark jobs upload <your_artifact_path>
```

**Example:**

```bash theme={null}
oleander spark jobs upload ./transformations/process_sales_data.py
```

**JAR example:**

```bash theme={null}
oleander spark jobs upload ./jobs/process_sales_data.jar
```

Every upload creates a new artifact version on the backend.

Upload options:

| Flag | Description |
| - | - |
| `--namespace` | Namespace used when uploading. Defaults to `default`. |
| `--py-files` | Local `.zip` or `.egg` dependency archive uploaded alongside a Python artifact |
| `--virtualenv` | Local virtual environment archive uploaded alongside a Python artifact |
| `--dry-run` | Show what would be uploaded without uploading anything |

### Delete a Spark artifact

```bash theme={null}
oleander spark jobs delete <artifact_name>
```

Use the exact uploaded artifact name, including the file extension.

**Example:**

```bash theme={null}
oleander spark jobs delete process_sales_data.py
```

### Submit a job

Submit an uploaded artifact to the oleander-managed cluster. Use the uploaded artifact name as the `entrypoint`.

```bash theme={null}
oleander spark jobs submit <entrypoint> --namespace <namespace> --name <run_name> --wait
```

**Example:**

```bash theme={null}
oleander spark jobs submit process_sales_data.py --namespace finance --name process-sales-data --wait
```

Common submit options:

| Flag | Description |
| - | - |
| `--cluster` | Cluster name. Defaults to the oleander-managed cluster when omitted |
| `--namespace` | Required job namespace |
| `--name` | Required job name |
| `--args` | Entrypoint arguments |
| `--sparkConf` | Spark configurations, for example `spark.default.parallelism=8` |
| `--packages` | Extra package coordinates |
| `--jobTags` | Job-specific tags |
| `--runTags` | Run-specific tags |
| `--wait` | Block until the job finishes |

Oleander-managed submit options:

| Flag | Description |
| - | - |
| `--driverMachineType` | Driver machine type |
| `--executorMachineType` | Executor machine type |
| `--executorNumbers` | Number of executor instances |

### Stop a running job

Abort a run that is currently in progress:

```bash theme={null}
oleander spark jobs stop <run_id>
```

The run ID is returned when you submit a job and is also visible in the oleander UI.

***

## Registered clusters

### List registered clusters

```bash theme={null}
oleander spark clusters list
```

## Registered EMR Serverless

Register your EMR Serverless cluster, then target it by name when submitting jobs.

### Register a cluster

```bash theme={null}
oleander spark clusters register <name> \
  --type emr-serverless \
  --region <region> \
  --account-id <aws_account_id> \
  --controller-role-arn <controller_role_arn> \
  --execution-role-arn <execution_role_arn> \
  --application-id <application_id> \
  --log-bucket <log_bucket>
```

See the [Spark documentation](/platform/compute/spark#registered-emr-serverless-spark) for IAM policy details.

### Submit a job

Use an EMR-compatible entrypoint, such as an S3 URI to a `.py` or `.jar` artifact.

```bash theme={null}
oleander spark jobs submit <entrypoint> \
  --cluster <cluster_name> \
  --namespace <namespace> \
  --name <run_name> \
  --wait
```

**Example:**

```bash theme={null}
oleander spark jobs submit s3://my-bucket/jobs/process_sales_data.py --cluster my-emr --namespace finance --name process-sales-data --wait
```

Additional EMR Serverless flags:

| Flag | Description |
| - | - |
| `--pyFiles` | Zip archive of Python dependencies |
| `--virtualenv` | Virtual environment archive for Python jobs |
| `--mainClass` | Main class for JVM jobs; use instead of Python-specific options such as `--pyFiles` and `--virtualenv` |
| `--executionIamPolicy` | IAM policy applied to execution |

## Registered Glue

Register your Glue cluster, then target it by name.

### Register a cluster

```bash theme={null}
oleander spark clusters register <name> \
  --type glue \
  --controller-role-arn <controller_role_arn>
```

See the [Spark documentation](/platform/compute/spark#registered-glue-spark) for IAM policy details.

### Submit a job

For Glue, the `entrypoint` is the Glue job name.

```bash theme={null}
oleander spark jobs submit <job_name> --cluster <cluster_name> --namespace <namespace> --name <run_name> --wait
```

Additional Glue flags:

| Flag | Description |
| - | - |
| `--workerType` | Glue worker type |
| `--numberOfWorkers` | Number of Glue workers |
| `--enableAutoScaling` | Enable Glue auto scaling |
| `--timeoutMinutes` | Timeout in minutes |
| `--executionClass` | Execution class, such as `STANDARD` or `FLEX` |
| `--executionIamPolicy` | IAM policy applied to execution |

<Note>
  For Glue jobs, `--args` must be passed as alternating key-value pairs, for example `--args source s3://bucket/in target s3://bucket/out`.
</Note>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.