spark-optimization Skill for Claude: What It Does and How to Install It

Makes Claude tune slow Apache Spark jobs: partitioning, joins, caching, shuffle and memory settings.

Installs: 10.9K; on skills.sh
Stars: 42.1K; on GitHub
Files: 1; besides SKILL.md
Updated: Oct 2026; last repo commit

This skill is a Spark performance guide for Claude, from wshobson's agents collection. SKILL.md covers the execution model (driver, job, stages, tasks), five performance factors, a quick-start session config and a list of do's and don'ts.

Worked examples are in references/details.md: seven patterns and a configuration cheat sheet. It loads when you optimize slow Spark jobs, tune executors, pick partitioning, debug skew or scale a pipeline.

Good for

Data engineers with slow or expensive Spark jobs
Teams scaling Spark pipelines to larger data
Anyone chasing data skew, spills or long shuffles
Developers choosing executor memory and partition sizes

Install spark-optimization

Claude Code

Copy this prompt.

Install the spark-optimization skill in Claude Code. Run this command:

/plugin marketplace add wshobson/agents
/plugin install data-engineering@claude-code-workflows

Then check that it loaded and tell me how to use it.

2
Paste it into Claude Code. It shows the change and asks before running it.

Pick one route. The plugin routes install the whole data-engineering plugin: 2 agents, 2 commands and 4 skills (counted from the plugin folder in the repo tree, which has no hooks or MCP config files). The skills-only routes install just this skill, with no agents or commands. The skill needs Spark in your own project. The examples use PySpark.

From the project README. Source.

What's in the skill

When it loads

Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

Sections in SKILL.md

When to Use This Skill
Core Concepts
1. Spark Execution Model
2. Key Performance Factors
Quick Start
Detailed patterns and worked examples
Best Practices
Do's
Don'ts

Key facts

Author: wshobson (repo owner)
License: MIT
Installs on skills.sh: 10,877 (Nov 2, 2026)
GitHub stars: 42,081 for the whole wshobson/agents repo (Nov 2, 2026)
Last repo commit: Oct 30, 2026
What it includes: 2 files including SKILL.md: references/details.md (no scripts)
Needs: A Spark setup of your own. The examples use PySpark, Parquet files, S3 paths and Delta Lake
Part of: The data-engineering plugin (version 1.3.3) in wshobson/agents. Counted from the plugin folder: 2 agents, 2 commands and 4 skills
Works in: Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot and Pi, per the README. Skills-only installers ask which tool you use

What it includes

Driver, job, stages, tasks, and five factors that slow Spark
Explains how a job splits into stages at shuffles and tasks per partition. Then lists shuffle, data skew, serialization, memory and partitions with a fix for each.
Try: “Why is this Spark job slow? Check for skew and shuffle problems.”
Read · Execution model and factors
Spark session with adaptive execution, Kryo and 200 shuffle partitions
An example session config, a Parquet read, a filter and aggregate, and a write to an S3 path in overwrite mode.
Write · Quick start session
Partition counts, repartition, coalesce, pruning (details.md)
Shows how to compute a partition count, repartition for even spread, coalesce without a shuffle, prune partitions and write partitioned output.
Write · Pattern 1: Partitioning
Broadcast, sort-merge, bucket and skew joins (details.md)
Covers four join types and a manual salting example for severe skew.
Try: “Rewrite this join so the small table is broadcast.”
Write · Pattern 2: Join optimization
Cache, storage levels, unpersist and checkpoint (details.md)
Explains when to cache a reused DataFrame, the storage levels, and when to unpersist or checkpoint a long lineage.
Write · Pattern 3: Caching
Executor memory, cores and memory fractions (details.md)
Walks through an 8 GB executor example, with the execution, storage and user-memory split.
Read · Pattern 4: Memory tuning
Cut shuffle size, pre-aggregate, approximate distinct (details.md)
Shows ways to shuffle less data, including coalesce instead of repartition and shuffle compression.
Write · Pattern 5: Shuffle
Parquet, column pruning, Delta Lake and Z-ordering (details.md)
Includes a Delta Lake write in overwrite mode and an OPTIMIZE ... ZORDER BY statement on a table path.
Write · Pattern 6: Data formats
Query plans, task metrics and skew checks (details.md)
Shows how to read explain output, stage task counts and per-partition row counts to find skew.
Read · Pattern 7: Monitoring
Production configuration template (details.md)
A reference block of Spark settings to start from.
Read · Configuration cheat sheet

From the official docs

Permissions

Low risk

What it can do

Read your data
Create and change things
Delete things
Read-only mode

Watch out for

Example writes overwrite their output path
SKILL.md ends its Quick Start with a write in overwrite mode to s3://bucket/output/, and details.md has a Delta write in overwrite mode. Swap in a test path before running either against real data. github.com
The plugin route installs more than this skill
The data-engineering plugin has 2 agents, 2 commands and 4 skills. We found no hooks or MCP config files in its folder. The skills-only routes install only this skill. github.com

Security advisories

No published security advisories as of Nov 2, 2026.

Technical details

Turning on read-only	No read-only mode. Use version control.
Sign-in and keys	SKILL.md and details.md do not mention any account or key.
Scopes	Claude can edit your Spark job code and config when you ask for tuning The Quick Start and Delta Lake examples write with mode overwrite to S3 paths The Delta Lake example runs OPTIMIZE with ZORDER BY on a table path

Questions

What does the spark-optimization skill do?

It gives Claude a Spark tuning playbook. Claude uses it to improve partitioning, joins, caching, shuffles and memory settings in your jobs.

How do I install spark-optimization?

Use the install box at the top of this page. Pick a skills-only route to get just this skill, or the plugin route to get the whole data-engineering plugin.

Is this skill from Anthropic?

No. It is in wshobson/agents, a community collection. The skill sits in the data-engineering plugin there.

Should I install the whole data-engineering plugin?

Only if you want the rest. The plugin folder holds 2 agents, 2 commands and 4 skills, and the repo tree shows no hooks or MCP config files in it.

Does it run my Spark jobs?

No. It bundles no scripts. It is a guide, and its do's list includes watching the Spark UI for skew, spills and GC while you run the job.

Does it work in the Claude app?

The install box has no Claude app upload for it, because the skill is built around PySpark code that runs in your own project. Use Claude Code or Codex. The README also lists Cursor, OpenCode, Antigravity CLI, GitHub Copilot and Pi.

Is it safe?

It bundles no scripts: just SKILL.md and one reference file. When you ask for changes, Claude edits your Spark code. The examples write to S3 paths in overwrite mode, so point them at test data first (our advice).

Information

updated Nov 2, 2026

Repository: wshobson/agents
Folder: plugins/data-engineering/skills/spark-optimization

wshobson/agents skills.sh

Sources

Docs, issue trackers and discussions this page is built from.

Show all 4

spark-optimization SKILL.md · github.com
wshobson/agents README · github.com
data-engineering plugin.json · github.com
spark-optimization references/details.md · github.com