This piece reviews the capability boundary of "building a high-quality Agent Skill (SKILL.md + bundled code) from scratch and running it long-term on GitHub." 150 research agents searched in parallel; 45 load-bearing conclusions each passed 3 independent adversarial verification perspectives (technical mechanism / product reality / recency), with verifiers instructed to default to "this doesn't hold unless you can confirm it," and to fetch first-hand sources wherever possible.
The verification actually did the work: it read agentskills.io/specification, Anthropic's official quick_validate.py, and the 17 official skills installed on the local machine, correcting a great deal of field names and numbers the draft had gotten wrong from memory. Grades: [Solid] = first-hand-sourced and survives three perspectives · [Narrowed] = the draft claim was overturned or heavily qualified, and what appears here is the corrected form · [Inference] = analytical derivation.
Recency: this round exhausted its web-search budget and had to rely on model knowledge (cutoff around early 2026) plus fetches of known authoritative URLs. February–July 2026 is an explicit blind spot, and Claude Code iterates fast (verification surfaced min-version gates from v2.1.129 through 2.1.218). Treat every version/feature/pricing-dependent claim as possibly stale and re-verify against the current version before publishing.
The recency disclaimer above has been cashed in: on 2026-07-25, Thariq (@trq212) of the Claude Code team published "The new rules of context engineering for Claude 5 models," filling the context-engineering slice of the blind spot. Four points directly affect the advice in this piece, all first-party:
- Lower the constraint density. For the Claude 5 generation (Opus 5 / Fable 5), the team removed over 80% of Claude Code's system prompt with no measurable loss on coding evals — hard rules that once guarded against worst cases are now handled by surrounding context and judgment. The same goes for a skill body; the official line: "avoid making them overconstrained, except in highly important areas." Section 3's genre still stands, but shrink the prohibition list to the few truly inviolable rules and leave the rest to the model.
- "Give examples" is demoted from best practice to anti-pattern. On the newest models, few-shot examples pin the exploration space; the official replacement is designing interfaces — parameter names and values are themselves the hint (the todo tool says everything through
pending/in_progress/completed). Put the expressiveness into your scripts' signatures, not into stacked examples. - Progressive disclosure got promoted from token-saving trick to the official mainline. For long skills the guidance is now explicit: split them across many files. Claude Code itself moved code review and verification into selectively loaded skills, and some tools are deferred-loading (definitions fetched via ToolSearch only when needed). Section 5's economics of references/ is confirmed first-hand.
- The official one-line positioning of skills: "lightweight guides" that "encode opinions, knowledge, or best practices particular to you, your team, or product." Don't write what the model already knows; write where you differ. Companion command:
/doctor(claude doctor) automatically rightsizes your skills and CLAUDE.md.
One row of section 8's boundary table needs amending too: memory has been split out of CLAUDE.md — Claude now saves memories automatically, and the old #-hotkey habit is obsolete; CLAUDE.md returns to lightweight: one line on what the repo is, tokens spent on gotchas, and never state what the model can see for itself.
The thesis up front: a high-quality skill has to win two contests at once. Building it means "in an environment with dozens of installed skills, the model reliably picks yours from a single line of description." Running it means "a stranger finds and trusts you on a shelf whose median install count is 1." The decider in both is not where you'd expect — not how cleverly the body is written, but the byte budget of the trigger surface, and the curation logic of the distribution shelf.
1. First, recognize the "spec vs Claude Code" split
This is the most fundamental cut; every later decision sits on top of it [Solid]. Agent Skills has a cross-vendor open specification (agentskills.io) that 40+ tools implement — Cursor, Gemini CLI, VS Code Copilot, OpenAI Codex — and Claude Code adds a large pile of private extensions on top. The two field sets differ, and getting it wrong drops you into the "runs locally, useless when shipped" trap.
The portable frontmatter is a whitelist of just 6 keys:
| Field | Constraint (spec) | Note |
|---|---|---|
name | required, 1–64, ^[a-z0-9-]+$, no leading/trailing hyphen, no -- | must equal the parent directory name (skills-ref enforces it); CC allows them to diverge, but keep them equal if you want portability |
description | required, 1–1024 chars, non-empty, no angle brackets | the line that decides triggering, see next section |
license | optional | keep it short: Apache-2.0 |
compatibility | optional, ≤500 | e.g. Requires git, docker, jq |
metadata | optional, string→string | see warning below |
allowed-tools | optional, marked Experimental | not a sandbox, see section 7 |
Claude Code adds ~14 private fields (when_to_use, paths, context, model, hooks, disable-model-invocation…), all outside the whitelist. Both reference validators (Anthropic's quick_validate.py, agentskills' skills-ref) report Unexpected key(s) for extra keys.
The easiest mistake to make: cramming Claude Code's private fields intometadata:for "compatibility." [Narrowed] This is mechanically ineffective — CC readscontext/modelfrom the top level, so moving them into metadata just makes things likecontext: forksilently fail, and the behavior no longer matches what your docs say. For open source, honestly write only the 6 keys; for CC-specific behavior, give a pasteable patch in the README.
2. description: the line that decides life or death
At startup, all that enters context is each skill's name + description (CC also appends when_to_use). The model has to match against this one line among "potentially 100+ skills" — it is the primary matching text for autonomous triggering. There are three length budgets you must not conflate:
- Spec / API / claude.ai: hard cap 1024 chars, rejected if exceeded.
- Claude Code runtime:
description+when_to_usecombined is truncated at 1536 chars, silently. - Global listing budget ≈ 1% of the context window. On overflow it does not truncate — it drops the description entirely, leaving only the name, starting with the skills you invoke least.
The third is the mechanism most fatal to open-source authors [Narrowed], and it forms a cold-start death spiral: a user has 40 skills installed, your new one has never been invoked → it's first to be demoted to name-only → there's no description to match against → it never auto-triggers → so it stays "least invoked." No description length can save it, because the whole thing is no longer in context.
Two actions that can save it: (1) put your strongest trigger keywords in the first 200 chars of the description, so it still matches even when cut to the front; (2) state explicitly in the README "first, invoke it once manually with /your-skill," to push it out of the "least invoked" queue.
How to write it (evidenced from Anthropic's own skills)
Third person is mandatory, but don't misread that as "only noun phrases allowed." Anthropic's best skills use an imperative addressed to the agent — Use this skill whenever the user wants to…. What's forbidden is the author's first person and the "You can use this to…" spoken to the end user (reason: the description is injected into the system prompt, and inconsistent point of view hurts discovery).
Length isn't a fixed range — it's a function of ambiguity. Measured across official samples, the median description length is 231 chars, not "one sentence":
- Trigger words are exclusive tokens (file extension
.pptx, a CLI name, a domain term) → safe up to ~900 chars, exhaustively listing verb phrases + extensions + bare keywords (pdfskill 437 chars,xlsx~950). - Trigger words are common nouns ("data," "report," "analysis") → even 200 chars becomes a global noise source.
The negative boundary is an optional third segment — don't add it by default. Only when your repo has ≥2 semantically adjacent skills and the poaching direction is clear should you write Do NOT trigger when… — because every skill's description is resident, so an exclusion clause taxes every request. That's exactly why only 4 of Anthropic's 17 skills use one.
If /skill-name works manually but it never auto-triggers, the description is the prime suspect but not the only one. Rule out in order: (1) frontmatter YAML parse failure (an unquoted colon or quote in the description makes the skill silently fail to load — only --debug shows it, and the symptom is identical to "not triggered"); (2) the task is too simple ("read this PDF" and other single-step tasks won't trigger regardless of description, because Claude only consults skills for tasks it can't easily handle itself); (3) CC tends to undertrigger, and the official prescription is to add a slightly "pushy" clause to the description.
3. The SKILL.md body: bimodal, not "the middle value"
Measured across Anthropic's 17 non-template SKILL.md files, line count is clearly bimodal [Narrowed], not a single target — with a gap between 129 and 236 lines:
- Behavioral-constraint type ~32–129 lines: rules that must be re-established every turn, kept short.
- Query-reference type ~236–541 lines: details pushed into
references/, body kept to navigation (theclaude-apiskill runs 541 lines).
Choose the tier by type; don't take the midpoint. The spec's <500 lines / <5000 tokens is a recommendation, not a hard reject (the claude-api skill is itself 68KB / ~17k tokens). But know the mechanism: Claude Code's auto-compaction keeps only the first 5000 tokens of each skill, re-attached skills share a 25000-token budget filled from the most recently invoked, and older ones are dropped entirely — so your inviolable rules must go at the top of the file.
Write in the voice of "standing instructions" — second-person imperative, present tense, no we'll / let's / in this guide (residue of narrative walkthroughs). The genre is "decision table + invariants + gotcha list + signposts," not prose: the first element after the H1 is best made a 3–7-row | Task | Approach | routing table (the Office-family convention), not an Overview that restates the description. Same information: giving the code directly ≈ 50 tokens, explaining-then-choosing ≈ 150 tokens — 3× the cost for zero gain. (Update 2026-07: the first-party line for the Claude 5 generation is “avoid over-constraining except in highly important areas” — keep the routing table and the gotcha list, trim prohibitions to the truly inviolable few; see the update box at the top.)
4. Scripts: don't decide a priori that "this should be a script"
The most counterintuitive finding [Solid]: don't decide off the top of your head which parts become scripts. The method: once the draft is written, take 2–3 real user-voiced prompts and spawn two subagents in the same turn — one with the skill, one without — then read the transcript and watch for the same helper the model still hand-writes in the with-skill arm; that is what gets frozen into a file in scripts/. Judgment tasks (selection/naming/layout/tone) always stay with the model. (This method only works in Claude Code; claude.ai has no subagents, so write a fallback.)
Landing conventions:
- Scripts interconnect via files + positional arguments, not natural language, not stdin; split into
analyze → plan.json → validate → apply → verify(the official plan-validate-execute pattern). Intermediate artifacts hit disk so the model iterates on the plan without touching originals. - Default to zero third-party dependencies, standard library only, invoke
python3; if you must depend, use PEP 723 inline metadata +uv run --script, and list dependencies explicitly at the top of SKILL.md. - cwd-independent: no package-relative imports, no relying on the current directory to find sibling resources — always
Path(__file__).resolve().parent. - Write operations non-destructive by default, offer
--dry-run, byte-identical output on re-run of the same input. - Error message = "what's wrong (including the value received) + the full set of valid values," optionally a third segment "which script to run next," with a non-zero exit code — so the model self-corrects with zero extra reads. The two-tier exit codes of
office/validate.py(argument errorexit(2)/ validation failureexit(1)/ pass0) are worth copying. For scripts where "stdout is the artifact," errors must go to stderr, or they pollute the downstream JSON.
5. references/: not a loader contract, just a naming convention
The only mechanism that gets a file read is a path string in the SKILL.md body that the model is willing to Read [Solid]. The directory name references/ itself has no magic. The corollaries are hard:
- A wrong path = a dead file, and the validator won't catch it (one well-known plugin shipped 19 broken links that survived 5 minor versions unnoticed). The spec's example filenames use uppercase
REFERENCE.md, but the repo actually has lowercasereference.md— copying the filename from the doc will failcatin a case-sensitive Linux sandbox, and you won't see it on macOS. - Each reference appears twice in SKILL.md: once on the line where the model makes a decision (with the trigger condition), once in a bottom resource index (with a one-line summary).
- References go only one level deep — nest two levels and the model may only
head -100-preview the referenced file and read partial information; a reference over 100 lines needs a table of contents at the top. - Criterion: content used with <50% probability in a single invocation goes into references/; but prohibitions always stay in the body, never sink. This is the real economics of progressive disclosure: the body is a recurring resident cost, a reference is a one-time cost.
6. Evaluation: two physically separate files
Trigger evaluation and execution evaluation must be separated, or you can't localize which layer failed [Solid]:
- Trigger eval (should it be selected): ~20 queries (8–10 positive + 8–10 near-miss negatives), each run 3 times for a trigger_rate, threshold 0.5, 60/40 train-holdout, and pick the description by test score, not train score. Negatives must be "just barely off," or the whole eval carries no information.
- Execution eval (completion once selected): run the same prompt twice (with skill / without), report all three deltas (pass_rate / time / tokens), and actively delete assertions that "both sides pass" — otherwise you're testing the model, not the skill.
There's exactly one tool worth using: the official skill-creator plugin (/plugin install skill-creator@claude-plugins-official), which bundles both eval types, grading, blind A/B, and an HTML review viewer. The "there is no built-in way to evaluate" line in the platform docs has fallen behind it. ⚠️ Two real traps: directory alphabetical order can flip the sign of the delta (name the primary dir new_skill/ to avoid it); with no timing.json, the tokens field silently becomes a character count — so each run's completion notification must be written to disk on the spot.
7. Security: allowed-tools is not a sandbox
The most common fatal misconception [Narrowed]: allowed-tools is a "per-turn skip-confirmation grant," not a capability restriction; it clears the moment you send your next message. disallowed-tools is also per-turn and Claude-Code-only. No frontmatter field is a sandbox. The only real security boundary is permissions.deny in settings.json (plus enterprise policy). To make a skill read-only, give the user a pasteable deny config in the README; don't rely on frontmatter.
The real ranking of supply-chain risk (calibrated across 137 real SKILL.md files on the local machine):
- A skill granting itself
allowed-tools(especiallyBash(*)) + payload in bundledscripts/— this is the number-one entry point. .claude-plugin/plugin.jsonloading the directory with agents/hooks/.mcp.json.- The
hooks:frontmatter. !`cmd`preprocessing (executes at render, before the model sees anything) — real-world prevalence ≈ 0, but once present it's zero-interaction execution.
A single master switch: add "disableSkillShellExecution": true to ~/.claude/settings.json. The core realization: installing a skill is, in threat-model terms, equivalent to installing executable software; the publisher's security responsibility = a machine-readable capability declaration + a reproducible audit checklist + a SECURITY.md written specifically for prompt injection.
8. Boundary: how to choose among skill / MCP / subagent / command / CLAUDE.md
This is the dimension where a wrong choice costs the most. First, kill an outdated mental model: slash commands have merged into skills — .claude/commands/deploy.md and .claude/skills/deploy/SKILL.md both produce /deploy and behave identically [Narrowed]. So the question is not "skill or command." The decision rule:
| Need | Pick | Why |
|---|---|---|
| Add a new tool / transport / auth (OAuth, long-lived connection, dynamic tool list) | MCP | Only MCP can conjure tools out of nothing; a skill can only orchestrate existing tools + its own scripts. And the two can ship in one repo (add a plugin.json to the skill dir) |
| Must happen every time (block push to main, must-run before commit) | hook | Both skills and CLAUDE.md are "requests," not "guarantees"; a hook is the enforcement. This is the most costly misclassification |
| A fact that must be true every session (build command, dir conventions) | CLAUDE.md | It's the only layer that survives /compact by being re-read from disk; a skill's "spans-the-whole-task rules" get silently truncated/dropped in long sessions |
| A multi-step flow that runs only when touching certain files | skill + paths: | skills support paths: globs too — don't force it into rules |
| A one-off task needing an isolated context | skill + context: fork | a subagent is a toggle on a skill, not a fifth parallel thing |
9. Distribution and cold start: the least feel-good dimension
Look at the evidence first [Narrowed]: install counts in the official directory are an extreme power law — one snapshot has 265 plugins, 3.6M installs, top10 = 53.8%, median = 1, and 52% of plugins have exactly 1 install. And the #1, frontend-design, is a 41-line SKILL.md with zero scripts. Two conclusions hold at once: code volume is not a condition for the head; and merely "getting on the shelf" is nearly worthless (8 of the top 10 are Anthropic-owned or big-vendor). What decides adoption is "being curated / brand," not "being listed."
The actionable landing:
- Two shelves:
claude-plugins-official(curated at Anthropic's sole discretion, no application process) +claude-community(reviewed, actually reachable by a solo dev — submit atplatform.claude.com/plugins/submit, lands in the community marketplace, pinned to a commit SHA, synced nightly). - Putting multiple harness manifests side by side in one repo = the cheapest 10× in distribution [Solid]. SKILL.md is already a cross-vendor standard across 40+ tools; not doing this is voluntarily forfeiting 90% of the shelves.
- For a one-line install, the repo root needs
.claude-plugin/marketplace.json, splitting one repo into multiple plugins (Anthropic splits its own into document-skills / example-skills / claude-api). The @ gotcha: in/plugin install <plugin>@<name>, the token after@is the marketplace.json top-levelname, not the repo name — copy the wrong one and the user gets "not found," while you can't reproduce it (because you already ranmarketplace add). Test both commands from a machine that has never added your marketplace. - README first screen: a one-line "what + who," a pasteable install, and a pasteable example prompt. A skill has no explicit invocation entry — a user who installs it and doesn't know what to say has effectively installed a broken thing.
10. Treating it as an open-source project: repo, versioning, CI
The minimum trustworthy set
- No LICENSE = exclusive copyright (others can't even legally copy it). LICENSE and CODEOWNERS are the only two community files that can't be defaulted from an org-level
.githubrepo; each must live in the individual repo. MIT vs Apache-2.0, the substantive difference: Apache adds an express patent grant (+ patent-retaliation termination), at the cost of a State-changes notice and a Trademark-use limitation. - CONTRIBUTING builds two separate gates: a syntactic gate (a vendored ~40-line validator — don't blindly depend on skills-ref, which self-declares "demo only") + an eval gate (every PR must include ≥3 trigger utterances + expected behavior). The latter is what converges the "taste wars" — it lets you reject "this is just a restated prompt with no scripts/references" on objective grounds.
- Issues use YAML forms that force capture of "client + version + skill version + the exact prompt used to trigger + whether it activated" — 90% of skill bugs are "didn't trigger" rather than "ran wrong," and a report missing the exact prompt is 100% unreproducible.
Versioning and releases
SKILL.md has no version field. What actually decides "can users receive updates" is the version in .claude-plugin/plugin.json — it's Claude Code's update cache key, and not bumping it means you didn't release [Solid, CONFIRMED]. Content-type semver defines breaking by the trigger surface + environment contract: description/when_to_use is append-only — never delete or narrow an existing trigger keyword (a deletion fails silently; the model simply stops loading the skill, no error). But be clear-eyed: semver on a bare skill is a communication convention, not enforcement (no channel resolves semver ranges); versioning is real only at the plugin layer. Git tags use {plugin-name}--v{version} (double hyphen); a bare v1.2.0 breaks resolution for plugins that depend on you.
CI (how a markdown-first repo makes meaningful gates)
- Never add a
paths:filter to a workflow that's a required status check — a skipped job never reports its check, and the PR is stuck forever at "Expected." For a small repo, just don't use paths; skill CI takes seconds. Matrix changes repeatedly break the required-check name, so make only one aggregation job (ci-ok,needs:[…]+if: always()) required. - Two co-primary checks: schema validation (name regex / no reserved words anthropic·claude / no XML tags, description ≤1024, body ≤500 lines — this decides whether the skill loads at all) + reference integrity (
lychee --offline+ a self-written script checking every path mentioned in SKILL.md exists). Keep reverse-reachability as a warning only (helper modules produce false positives). - Choose DCO over CLA: Apache-2.0 clause 5 already fixes inbound=outbound; CLA Assistant needs
pull_request_target+ a write-scoped PAT — exactly the attack surface you avoid everywhere else.
11. Governance and metrics
Three findings passed all three perspectives unanimously [Solid, CONFIRMED]:
- PR triage rests on reproducible behavioral evidence, not the text diff: the template forces a trio (the trigger prompt verbatim / before-after transcript / whether the description changed), plus a
needs-evidencelabel + 14-day auto-close. Without this, the queue is 100% going to die. - The default answer for scope control is "no new skill directory": write the Non-goals into the README first screen and CONTRIBUTING, route every new skill to fork / self-hosted marketplace rather than merge, and close politely on the spot in 1–2 sentences.
- SLA = first triage within 48 hours (not review completion), with a public 7-day reply commitment in the README; one fixed ≤90-minute triage window per week; stale-bot at 30 days stale / 14 days close (don't copy k8s's 90 — a skill's lifecycle is too short).
Metrics must separate vanity from signal: the main criterion is the with/without pass_rate delta (not the only one — the tool_calls cost delta, blind-eval winner, and trigger accuracy each carry independent information). Stars are nearly decoupled from real use; a bare skill has no download registry, so of the four common health ratios only two (merged-PR/star, issue-close rate) are directly computable, and getting download numbers means additionally packaging as a plugin/npm. A Release asset's download_count is the cleanest install proxy (public, zero-PII, cumulative, never lost). Never embed a covert curl beacon in SKILL.md or scripts — it runs in someone else's agent session; this is a consent issue, not a technical one.
One last reality you'll inevitably hit: an AI-generated PR flood is the default fate of a skill repo (Anthropic's own skills repo has 1,023 PRs, only 44 merged, 742 hanging open, and no CONTRIBUTING at the root). Install gates early, with interaction-limits as a temporary rate-limit rather than a permanent close.
12. What this research did not cover
- February–July 2026 is a complete blind spot. Anything touching Claude Code versions / feature names / pricing, and the plugin-and-marketplace layer (a surface less than a year old), may already have changed — verify against the current version before publishing. (2026-07-27: the context-engineering slice has since been filled first-party — see the update box at the top; the plugin/marketplace surface remains uncovered.)
- Anthropic's internal assets are invisible. Whether it has skill usage telemetry, internal eval sets, or the curation logic of the skill ecosystem can't be confirmed — this piece's adoption judgments cover only the public domain.
- Many numbers are the "as-claimed-in-docs/source" current values, not long-term contracts. This piece captured the 2026-07-24 implementation;
skills-refself-declares "demonstration purposes only," the spec page carries no version number, and governance has been handed to the community repoagentskills/agentskills. Pin a commit when citing specific script names and directory paths; don't treat them as a stable API.
Main sources (all passed three-perspective verification, all first-hand): the Agent Skills open specification at agentskills.io/specification (6-key frontmatter whitelist, name/description/compatibility constraints); Anthropic's anthropics/skills repo — skills/skill-creator/scripts/quick_validate.py (the ALLOWED_PROPERTIES set, angle-bracket rejection, name-length cap), skills/pdf|docx|xlsx|pptx/SKILL.md (real description lengths and genre, license: Proprietary), scripts/office/validate.py (two-tier exit codes), the eval toolchain run_eval.py/run_loop.py/aggregate_benchmark.py; agentskills/agentskills's skills-ref/src/skills_ref/validator.py (name==parent-dir enforcement, the demonstration-only disclaimer); Claude Code's official docs at code.claude.com/docs/en/skills (17-field frontmatter table, 1536-char listing truncation, the 1% listing budget and demotion, 5000/25000-token compaction, allowed-tools per-turn grant semantics, disableSkillShellExecution, slash commands merged into skills); platform.claude.com's Agent Skills overview and best-practices (plan-validate-execute, progressive disclosure, the three-step eval, one-level-deep references); a local telemetry snapshot of the official plugin directory (265 plugins / 3.6M installs / median 1 / top10 = 53.8%); GitHub community-health-file docs, choosealicense (the MIT vs Apache-2.0 permission tables), Keep a Changelog, and CNCF/Node.js governance docs. 45 load-bearing conclusions entered three-perspective adversarial verification (8 unanimous / 37 partly corrected / 0 refuted); this piece presents the corrected versions, with grades and recency risk labeled throughout. Source added 2026-07-27: Thariq (Claude Code team), “The new rules of context engineering for Claude 5 models,” x.com/trq212, 2026-07-25.