Skip to content
Apothem

Contributing to Apothem — Developer Guide

Maintain and extend the Apothem ecosystem by authoring rules, commands, helpers, skills, and hooks with canonical schemas, frontmatter requirements, and validation procedures.

This guide helps developers maintain and extend the Apothem ecosystem with SOTA quality standards.

Quick Reference: Artifact Types

This guide covers five artifact types, each with a specific purpose. Rules, commands, helpers, and skills carry YAML frontmatter; hooks are registered in hooks.json and settings.json and carry none. Output styles are a further artifact class with two required frontmatter keys, name and description; see the Frontmatter Schema.

ArtifactPathPurposeWhen to Create
Rulesrc/apothem/rules/*.mdBehavioral mandate or operational directiveNew principle, policy, or governance mandate established
Commandsrc/apothem/commands/*.mdSlash command (/command-name) for complex workflowsNew multi-step user workflow
Helpersrc/apothem/agents/*.mdPersistent delegated-worker definition for parallel tasksNew recurring specialized task (audit, explore, quality-check)
Skillsrc/apothem/skills/{name}/SKILL.mdReusable technique with detection signalNew detectable, reusable technique pattern
Hooksrc/apothem/hooks/ + hooks.json + settings.jsonEvent-triggered validation or state managementNew lifecycle event or validation need

Harness Adapter Contract

Harness identity and capability facts live in src/apothem/lib/harness_registry.py. A harness change is complete only when these surfaces agree:

SurfaceRequired update
Registry rowPublic ID, package key, entry point, scope, output format, target paths, docs paths, capability file, standard pin, fixture tests, package-data key, template sources, and capability statuses.
Adapter package__init__.py, install.py, update.py, uninstall.py, verify.py; add materializer.py only when the harness renders a native config file.
Propagation manifestTemplate and cohort paths for manifest-backed adapters.
Capabilitiessrc/apothem/harnesses/<package>/capabilities.yml with every required capability and rationale for unsupported or discovery-pending states, plus a conversion_losses map for each converted commands, agents, or rules cohort (see the harness registry).
Convention pinSTANDARD-CONVENTION-PIN.md with vendor URL, snapshot date, canonical filename/schema, evidence reference, and unsupported capability rationale.
TestsRegistry parity, adapter fixture test, golden install-plan row, dry-run/no-write behavior, idempotence, and package-data coverage.
DocsHarness reference page, comparison page, capability/MCP notes, and examples that use fake placeholders.

Project-scope adapters must set requires_project = True, accept project: Path | None on lifecycle methods, and reject missing or unsafe project roots before writes. User-scope adapters must keep paths under the harness's documented user configuration root.

Golden Fixture Policy

tests/fixtures/harness-golden-plans.json records the public install-plan shape for all seventeen registry harnesses: public ID, materialization mode, source cohort/template path, and normalized target path under <ROOT>. When registry, manifest, or template target behavior changes intentionally, update the golden fixture in the same commit as the source change so the diff shows the behavior delta.

Validation Before Write

The shared install driver validates every source and target before the first write. It rejects missing manifest sources, target traversal, target paths outside the selected root, and symlink crossings. Dry-runs execute the same validation and return structured skipped or unchanged results without creating files.

Redaction and Example Data

Profile diagnostics redact token, secret, password, credential, API key, private key, authorization, auth, and header-shaped fields. Documentation, fixtures, examples, and JSON snippets use example.invalid, Example User, example-user, and obvious placeholders instead of real-looking secrets or personal accounts.

Before You Write: The Canonical Schema

The machine-readable, gate-enforced contract is the per-class JSON Schemas under src/apothem/schemas/*.schema.json — additionalProperties rejects unknown keys, so the conformity gate is the final word. Two human-readable projections track them: site/content/docs/reference/frontmatter-schema.mdx (the required-key index) and site/content/docs/reference/artifact-schema.mdx (per-class templates and worked examples). Where a page disagrees with the JSON Schemas, the schemas win.

  • Read the JSON Schema for your artifact type under src/apothem/schemas/
  • Read site/content/docs/reference/artifact-schema.mdx § "Schema" for the template and a worked example
  • Copy the required fields, fill in mandatory values, add optional fields as needed

Frontmatter Quick-Check

Every artifact frontmatter must pass these checks. Apply the version, updated, and portability checks only to a class that carries the key: rules carry none of them, and skills carry no portability.

✓ Valid YAML syntax (use tools: `yamllint` or PowerShell `ConvertFrom-Yaml` where available)
✓ All mandatory fields present (per schema for artifact type)
✓ name: uses kebab-case
✓ version: semantic MAJOR.MINOR.PATCH
✓ updated: ISO 8601 date (YYYY-MM-DD)
✓ description: single-line, non-empty
✓ scope fields valid per artifact type (rules: pathFilter + alwaysApply; CLAUDE.md root: scope)
✓ portability: string, for example `universal` (the schemas do not enumerate its values)
✓ No typos in field names — must match canonical schema exactly
✓ Optional fields use correct enum values

Naming Conventions

Artifact Names (kebab-case)

  • Rules: operational-mandates, clean-room-generation, context-management
  • Commands: plan-execute, plan-spec, plan-generate
  • Helpers: codebase-explorer, convention-auditor, quality-gate
  • Skills: plan-suite, ecosystem-audit

Pattern: lowercase, hyphens between words, no spaces or underscores.

File Naming

  • Rules: {name}.md
  • Commands: {name}.md
  • Helpers: {name}.md
  • Skills: Inside folder src/apothem/skills/{name}/SKILL.md (exact case: SKILL.md)
  • Hooks: Inside folder src/apothem/hooks/. All event handlers are Python. Every hook command in the Claude Code settings.json template uses exec form: command is the interpreter placeholder ${PYTHON_BIN}, resolved at install time, and args holds the script path and its arguments. That template runs ${HARNESS_ROOT}/.apothem/support/hooks/dispatch.py <event> [<message-name>] and ${HARNESS_ROOT}/.apothem/support/conformity/gate.py --hook by absolute path, so the installed copy needs no importable apothem package. The Claude Code plugin registers hooks in src/apothem/hooks/hooks.json instead, as shell commands that run hooks/lib/bootstrap.sh through bash; that stub locates Python and runs dispatch.py, so the handlers stay Python. Register a new hook in hooks.json and in the Claude Code settings.json template. dispatch.py routes SessionStart to session_start_bootstrap.main() and every other whitelisted event to emit_hook_context.main(). The only shell stubs permitted under src/apothem/hooks/ or scripts/ are the six locator and bootstrap files (find-python.{ps1,sh}, find-pwsh.{ps1,sh}, bootstrap.{ps1,sh}); any other .ps1 / .sh file fails the scripts/dev/validate_hooks.py hygiene check.

Cross-References

When referencing other artifacts, use exact names:

  • Rules: "operational-mandates" (name field value)
  • Commands: /plan-execute (with slash for human docs; without in code)
  • Helpers: codebase-explorer (name field value)
  • Skills: plan-suite (name field value)

Never use file paths or truncated names in prose.

Portability: Windows vs. Unix

Every command MUST explicitly declare its portability status in the portability field. A helper MAY declare it. Skills, rules, and output styles carry no portability field.

Universal

  • Works identically on Windows, macOS, and Linux
  • No shell assumptions
  • Paths use forward slashes (/)
  • No hardcoded absolute paths
  • No Windows-specific PowerShell syntax
portability: "universal"

Universal with Windows Caveats

  • Universal across POSIX hosts; the named caveat applies on the Windows platform variant declared in the content
  • Document the caveat explicitly in content
  • Example: "Symlinks not supported on Windows before Win 10"
portability: "universal-with-windows-caveats"

Document the caveat in a note immediately after the title.

Unix-Only / Windows-Only

  • Explicitly restricted to one platform
  • Rare in Apothem — most should be universal
  • Use only if platform-specific features are essential
portability: "unix-only"

Document the reason in the artifact's first section.

Scope: Always-On vs. Path-Filtered

Always-On

Rule applies everywhere. Five rules are always-on: agnostic-posture, authority-inquiry, session-closure, context-management, and the rule index (apothem-rule-index). The set is small because every always-on rule costs context in every session.

pathFilter: ""
alwaysApply: true

Path-Filtered

Rule applies conditionally based on file path.

pathFilter: "**/*.py, **/*.toml"
alwaysApply: false

The pathFilter uses glob patterns (same format as .gitignore). When a file matches the filter, the rule activates.

Version Semantics

Each versioned artifact carries its own semantic version (MAJOR.MINOR.PATCH). Commands and skills require it, helpers and output styles MAY carry it, and rules carry none:

  • PATCH — a behavioral change that corrects an existing directive without breaking the contract.
  • MINOR — additive non-breaking changes: new optional sections, new optional frontmatter fields, new examples. Existing consumers continue to work without change.
  • MAJOR — breaking schema or contract changes: removed sections, renamed fields, altered behavior of an existing directive, changes that require downstream artifacts to adapt.

Typo fixes, link fixes, formatting, and comment-only or documentation-only edits leave the version unchanged. Only a behavioral change bumps it. Such an edit still updates the artifact's updated date.

Examples:

  • vX.Y.Z → vX.Y.(Z+1) (PATCH: a corrected directive that changes behavior)
  • vX.Y.Z → vX.Y.Z (no bump: typo or link fix; updated changes)
  • vX.Y.Z → vX.(Y+1).0 (MINOR: added a new optional section)
  • v0.Y.Z → v(0+1).0.0 (MAJOR: declared contract stable; new artifacts seed at v0.Y.Z)
  • vX.Y.Z → v(X+1).0.0 (MAJOR: removed mandatory frontmatter field)

Bump version and updated together on every behavior-changing edit to an artifact that carries them. Append a matching row to the ecosystem-level CHANGELOG.md for releases that bundle artifact changes.

Plan-suite revisions

The artifact rule above does not apply to plan suites. A plan suite's version is the vX.Y.Z value recorded in its MASTER-PLAN.md header and Revision History table. A typo fix to a plan suite is a revision that bumps the PATCH number. Step 10.2 of /plan-review sets the classification and appends the Revision History row.

  • vX.Y.Z → vX.Y.(Z+1) (plan suite, typo fix: PATCH)
  • vX.Y.Z → vX.Y.Z (artifact, typo fix: no bump; the artifact rule above)

When to Edit an Artifact

Rules

  • Treat the Mandatory/Optional structure as load-bearing — restructuring it ripples through every dependent artifact.
  • Refine Anti-Patterns, Seriousness Scaling examples, and citations as understanding deepens.
  • Capture the rationale for any structural change in the relevant memory topic file.

Commands

  • Argument shape is part of the public contract — break it only with explicit intent and propagate the change to every caller (MAJOR bump).
  • Keep workflow steps and return contracts tight and current.
  • Document non-obvious step sequencing inline.

Helpers

  • Keep disallowedTools minimal-but-adequate; document the rationale when it changes.
  • Refine Operating Principles and Return Contracts as the helper's role matures (MINOR bump).
  • Track capability shifts inline so callers can recalibrate expectations.

Skills

  • Keep Procedure steps and Examples synchronized with reality.
  • Detection signals must reflect actual user phrasing in current usage.
  • Split a skill once it crosses ~150 lines (see auto-memory.md § Topic File Management).

Step-by-Step: Adding a New Rule

1. Assess Orthogonality

Does this rule overlap with existing rules? Search src/apothem/rules/ for related content.

If overlap >50%, consider extending an existing rule instead. See persistent-conventions-vigilance.md § Artifact Evolution.

2. Write Content

Follow the structure:

  • Title: # Rule: {Title}
  • Purpose section
  • Obligations / Core directives (numbered, linked)
  • Seriousness Scaling table (always required)
  • Anti-Patterns section
  • Enforcement section

3. Create Frontmatter

Use the schema from site/content/docs/reference/artifact-schema.mdx § Schema: Rules. Set:

  • name: kebab-case identifier
  • description: one-line summary for registries
  • pathFilter: glob list for path-filtered rules; the empty string ("") for always-on rules
  • alwaysApply: true for always-on rules, false for path-filtered rules

Rules carry only these four fields — no version / updated / scope / portability / implements.

A new rule is path-filtered. A new always-on rule needs the core list and the byte ceilings raised in the same change: the list in tests/unit/test_always_on_core.py and the ceilings in conformity/always_on_budget_grep.py. A companion rule (<parent>-<aspect>) takes its parent's pathFilter, or a narrower one.

Example:

---
name: "my-new-rule"
description: "Brief one-liner"
pathFilter: "**/*.py"
alwaysApply: false
---

4. Update the Rules README

Add a row to the table for the rule's family in the Rule families section of src/apothem/rules/README.md:

| `my-new-rule.md` | One-line concern of the new rule. |

Pick the family table whose concern matches the rule. Also name the rule in the line of src/apothem/rules/apothem-rule-index.md for the situation that calls for it; tests/unit/test_always_on_core.py fails when a rule is missing from the index.

5. Append to CHANGELOG.md

Add a row under the next release's ### Added section describing the new rule.

6. Run Validation

Invoke the ecosystem-audit skill to verify the new rule, focusing on the rules area:

Invoke the ecosystem-audit skill with --focus rules --fix

Verify no errors reported for your new rule.

Step-by-Step: Adding a New Command

Commands are slash commands like /plan-spec, /plan-execute, /security-audit, etc.

1. Determine Role

Commands are user-triggered workflows. The model runs one on its own only when its disable-model-invocation setting is false (step 2). Ask:

  • Is this a workflow the user types /command-name?
  • Does it orchestrate multiple steps?
  • Could it timeout or require user input mid-execution?

If "no" to any of these, it's probably a skill, not a command.

2. Create File

Path: src/apothem/commands/{name}.md

Frontmatter (from site/content/docs/reference/artifact-schema.mdx § Schema: Commands):

---
name: "my-command"
version: "X.Y.Z"
updated: "2026-04-20"
description: "What this command does"
argument-hint: "[path/to/input] [--flag VALUE]"
disable-model-invocation: false
portability: "universal"
allowed-tools: "Read, Glob, Grep"
---

disable-model-invocation: true makes a command operator-invoked only; with false, the model may run it when a request matches its description. The default is false for read-only and advisory commands, with a description that leads with the use case; 35 of the 45 commands use it. A command stays true when it changes state outside the working tree or depends on the operator choosing the moment (10 commands). The invocation table in src/apothem/commands/README.md lists every command's setting; add a row for a new command. Keep allowed-tools read-only: a harness that honors it runs those tools without asking. Grant a side-effecting tool only as a scoped rule, such as Bash(git log *); the schema rejects "*" and any of Bash, PowerShell, Write, Edit, NotebookEdit, or WebFetch written without a scope.

3. Write Content

Structure:

  • Title: # /{name} — Human-Readable Title
  • Role section (who executes this)
  • High-level instructions
  • Workflow section with steps
  • Return contract / output format

One harness's skill emission splits each command into a slim SKILL.md plus references/ files by H2 section. Give a command one ## Workflow section, close every code fence, and classify any new H2 kind in src/apothem/lib/claude_skill_split.py; tests/unit/test_claude_skill_split.py fails on an unclassified kind. The rules are in the Skill-emission split contract of src/apothem/commands/README.md.

4. Update the README and CHANGELOG.md

Add a row to the table for the command's section in src/apothem/commands/README.md, in addition to the invocation-table row from step 2; add a CHANGELOG row under the next release's ### Added section.

5. Validate

Invoke the ecosystem-audit skill, focusing on commands:

Invoke the ecosystem-audit skill with --focus commands --fix

Step-by-Step: Adding a New Helper

Helpers are persistent delegated-worker definitions for parallel work.

1. Identify Pattern

Does this helper fit one of the team patterns listed below?

  • Research Team: read-only, information gathering
  • Audit Team: verification, consistency checks
  • Implementation Team: parallel code changes
  • Quality Team: lint, test, type-check, security
  • Documentation Team: doc updates
  • Other: specialized task

2. Create File

Path:

src/apothem/agents/{name}.md

Frontmatter:

---
name: "my-helper"
version: "X.Y.Z"
updated: "2026-04-20"
description: "What this helper specializes in"
tools: "Read, Glob, Grep, Bash"
disallowedTools: "Write, Edit"
maxTurns: 15
portability: "universal"
---

Leave out model, effort, and permissionMode so the helper inherits the session's choices (the agnostic posture), and leave out memory unless the helper needs a persistent-memory scope (user, project, or local).

3. Write Content

Sections:

  • Operating Principles (read-only, evidence-based, binary verdicts, exhaustive)
  • Workflow (numbered steps)
  • Return Contract (max tokens, structure, required fields)

4. Update the README and CHANGELOG.md

Add a row to the Index table in src/apothem/agents/README.md; add a CHANGELOG row under the next release's ### Added section.

5. Test

Deploy the helper in a real scenario and verify it works as documented.

Step-by-Step: Adding a New Skill

Skills are reusable, detectable techniques.

1. Identify Reusability

Before writing a skill:

  • Have you seen this problem/technique 3+ times?
  • Is it detectable? (Does a user request pattern signal it?)
  • Can you codify it reproducibly?

If all three are yes, write a skill.

2. Create Structure

%%{ init: { "theme": "neutral" } }%%
%% verified: 2026-04-27 %%
%% provenance: site/content/docs/reference/artifact-schema.mdx (skill artifact schema) %%
%% cross-reference: src/apothem/skills/plan-suite/SKILL.md (canonical example) %%
flowchart TD
    accTitle: New skill directory structure
    accDescr: Top-down flowchart of the directory structure for a new skill under src/apothem/skills: a {skill-name} folder holding the required SKILL.md plus optional examples and templates subfolders.
    SKILLS["src/apothem/skills/"]
    SKILLS --> NAME["{skill-name}/"]
    NAME --> SKILLMD["SKILL.md<br/>(main definition · required)"]
    NAME --> EX["examples/<br/>(optional · example workflows)"]
    NAME --> TPL["templates/<br/>(optional · reusable templates)"]

3. Write SKILL.md

Frontmatter:

---
name: "skill-name"
version: "X.Y.Z"
updated: "2026-04-20"
description: "What this skill teaches"
archetype: "workflow-template"
userInvocable: true | false
disable-model-invocation: true | false
allowed-tools: "Read, Glob, Grep"
argument-hint: "[args]"               # optional; if userInvocable
---

Content structure:

  • Purpose
  • When to Use This Skill
  • Prerequisites
  • Step-by-Step Procedure
  • Common Mistakes
  • Examples

4. Update the README and CHANGELOG.md

Add a row to the Index table and a row for each of the skill's files to the Folder contents table in src/apothem/skills/README.md; add a CHANGELOG row under the next release's ### Added section.

5. Validate

Invoke the ecosystem-audit skill, focusing on skills:

Invoke the ecosystem-audit skill with --focus skills --fix

Validation Checklist Before Committing

  • Frontmatter passes YAML syntax check
  • All mandatory fields present per schema
  • name field uses kebab-case
  • version (versioned classes only; rules carry none) is semantic (MAJOR.MINOR.PATCH) and was bumped if behavior changed
  • updated matches today's date for any artifact you touched that carries the field
  • portability field set and valid (required on commands, optional on helpers)
  • CHANGELOG.md carries the change in the appropriate category under the next release's section
  • No plan-internal references (CM-7 compliance)
  • Cross-references are consistent with other artifacts
  • New artifact registered in its folder README index
  • Artifact tested (if applicable)
  • ecosystem-audit skill runs with no errors
  • Content follows structural conventions (title format, sections, etc.)

Referencing Other Artifacts

Use these exact conventions:

Reference a Rule

See `src/apothem/rules/operational-mandates.md` or shorthand: the Operational Mandates rule.
In prose: ... per CM-1–10 (Operational Mandates rule) ...

Reference a Command

See `/plan-execute` command. Or: Run `/plan-execute`.
In prose: The `/plan-execute` command handles phase execution.

Reference a Helper

Deploy the `codebase-explorer` delegated worker.
In prose: The codebase-explorer worker finds patterns.

Reference a Skill

Use the `plan-suite` skill.
In prose: The plan-suite skill provides the Master Plan Suite template.

Reference CLAUDE.md Sections

See CLAUDE.md § Harness Adapter Pattern.
Or: § Plans Discipline.

Linting and Format

All artifacts use:

  • Markdown: GFM (GitHub Flavored Markdown)
  • YAML: YAML 1.2 compliant
  • Line endings: LF (Unix-style)
  • Encoding: UTF-8
  • Line width: Soft wrap at 100 characters for content (no hard breaks)

Use the Ruff formatter / linter for Python files. For Markdown, use prettier or equivalent.

Testing and Measurement

Run the suite

python -m pytest                      # unit, integration, property, conformity, scripts
python -m pytest tests/property       # Hypothesis properties (profile loader, frontmatter, merge)

The suite is hermetic: it passes with the checkout under $TMPDIR and with a python on PATH that lacks the CLI prerequisites, because the integration tests run the interpreter that runs pytest and tokenize nested temp and source paths longest-first.

Coverage per package

CI measures three packages and holds each to its own floor. None of them is folded into another.

PackagePathsFloorWhere the floor lives
enginecli, harnesses, lib, hooks, statuslines80[tool.coverage.report] fail_under
conformitysrc/apothem/conformity81[tool.coverage.package-floors]
auditsrc/apothem/audit70[tool.coverage.package-floors]

The figure is line plus branch coverage, the same percentage coverage.py reports with branch = true; the metrics artifact also records the statement-only figure under coverage_lines. Each package floor is the measured line plus branch value rounded down. Raise a floor when coverage rises; never lower one to make a change pass. To check locally:

python -m pytest --cov=src/apothem --cov-report=json:coverage.json
python scripts/dev/collect_metrics.py --coverage-json coverage.json --check-floors \
  --skip-hooks --skip-coldstart --harness zed

The metrics artifact

The CI metrics job runs scripts/dev/collect_metrics.py on every push and pull request and uploads metrics.json (schema_version: 1). Every value that is null carries its reason under nulls.

KeyWhat it measures
coverage.engine, coverage.conformity, coverage.auditLine plus branch percent per package, with coverage_floors.
always_on_bytes[harness], always_on_files[harness]For each of the 17 harnesses, after profile init and install --harness <id> into a scratch HOME and project: bytes on disk of the instruction files that harness loads in full at launch.
cli_coldstart_msMedian of python -m apothem --version (with --help under cli_coldstart_detail_ms).
hook_e2e_ms[event], hook_injected_chars[event]Per registered event and matcher (PreToolUse:Write, Stop, …): the median total time of the hook chain on a real payload, and the context characters it injects.

Which files count as loaded at launch is a per-harness rule taken from each vendor's documented instruction surfaces. Listings that carry only a description (skills, commands, subagents) are not counted. The rule text is also written into the artifact under rules.always_on_bytes:

antigravity     ~/.gemini/{GEMINI,AGENTS}.md, ~/.gemini/config/{GEMINI,AGENTS}.md,
                ~/.gemini/config/rules/*.md with trigger: always_on, project
                GEMINI.md and AGENTS.md. The plugin rules/ folder is not a
                documented launch surface.
claude-code     ~/.claude/CLAUDE.md and ~/.claude/rules/**/*.md without paths:
                frontmatter, plus the project equivalents.
codebuddy       .codebuddy/rules/**/*.md (project and home) unless
                alwaysApply: false, plus CODEBUDDY.md.
codex           ~/.codex/AGENTS.md and the project AGENTS.md.
cursor          .cursor/rules/**/*.mdc with alwaysApply: true, and AGENTS.md.
gemini-cli      ~/.gemini/GEMINI.md and the project GEMINI.md.
github-copilot  .github/copilot-instructions.md and the project AGENTS.md.
hermes          ~/.hermes/SOUL.md, project .hermes.md, AGENTS.md, CLAUDE.md.
                The support tree and apothem/rules/ are not launch surfaces.
kimi-code       ~/.kimi-code/AGENTS.md, project AGENTS.md and .kimi-code/AGENTS.md.
kiro            .kiro/steering/**/*.md with inclusion: always or no inclusion
                key, and the project AGENTS.md.
open-claw       Workspace bootstrap files ~/.openclaw/workspace/*.md.
opencode        ~/.config/opencode/AGENTS.md, the project AGENTS.md, and every
                file the instructions globs in opencode.json match.
qwen-code       The context files named by context.fileName in
                ~/.qwen/settings.json (default QWEN.md), in ~/.qwen/ and the project.
trae            .trae/rules/**/*.md with alwaysApply: true, and
                ~/.trae/user_rules/*.md.
windsurf        .devin/rules/**/*.md and .windsurf/rules/**/*.md with
                trigger: always_on, the global global_rules.md, and AGENTS.md.
zed             The first of .rules, .cursorrules, .windsurfrules, .clinerules,
                .github/copilot-instructions.md, AGENT.md, AGENTS.md, CLAUDE.md,
                GEMINI.md in the project, plus ~/.config/zed/AGENTS.md.
glm             Nothing: GLM is a model backend and loads no instruction files.

Behavioural evals

The suite under evals/ measures behaviour rather than form: 148 trigger cases (each model-invocable command, skill and subagent fires on a request in its domain and stays quiet on a near miss, and six user-invoked-only commands do not start on their own), 32 outcome cases (one per plan stage, research stage and audit dimension) and 6 rule-effect pairs (a sampled rule's text on and off). The cases are plain data validated for free by python -m pytest tests/unit/test_eval_suite.py; running them is billed and manual. See Run the behavioural evals for the Evals workflow, a local run and how to add a case. After editing a rule that has a pair, run python scripts/dev/sync_eval_rule_cases.py.

Benchmarks

make benchmarks runs the per-class drivers under src/apothem/benchmarks/, except bench_agents.py. Each driver exits 0 pass, 1 over budget, 2 error (the measured run did not execute), or 3 not measured.

  • bench_hooks.py runs every shell: bash command that hooks/hooks.json registers, as written, on a real payload per event and matcher, and reports the median total time, the slowest command (checked against the event's budget) and the characters injected. A command that cannot run as registered (exit 126 when a script lacks its executable bit) is an error, not a pass.
  • bench_agents.py reports NOT MEASURED (exit 3): a subagent spawn needs a host harness and a model call. make benchmarks does not run it.

The always-on budget check (always-on-budget-grep) counts whitespace-separated words with the Bindings section and companion pointer lines excluded. Its JSON report also carries always-on-loaded: the bytes, characters and words of the frontmatter-stripped bodies a harness loads, summed over the always-on rules. Pointed at the shipped rules directory, the check also fails when that sum exceeds a byte ceiling (reported as always-on-ceiling-bytes) or the rule index exceeds its own. Claude Code loads its rule copies without the Bindings section, so its loaded size is smaller than the figure this check reports.

Ecosystem Audit: Running Validation

Two complementary validation surfaces cover the ecosystem:

Machine-checkable (Python validators) — run before every commit:

python scripts/dev/validate_hooks.py       # Hook structure + Python-only hygiene
python scripts/dev/validate_ecosystem.py   # Full tree + frontmatter + delegated hook checks
python scripts/dev/chaos_pass.py           # Adversarial sweep across configured hooks
pytest tests                                    # Unit tests for the hook runtime

Assistant-driven — for cross-reference, narrative, and convention audits:

/ecosystem-audit --focus all --fix

This runs in parallel:

  • Structure audit: Registry accuracy, file existence, frontmatter validity
  • Artifact audit: Cross-references, mandate coverage, pipeline ordering
  • Memory audit: Memory file claims vs. filesystem state

Expected output: PASS with zero findings from both surfaces.

If FINDINGS reported:

  • Review each finding's severity (Critical / Important / Advisory)
  • Address Critical items immediately
  • Plan Important items for next cycle
  • Consider Advisory items for next iteration

Questions?

Refer to:

  • Frontmatter schema: site/content/docs/reference/artifact-schema.mdx
  • Artifact lifecycle: src/apothem/rules/persistent-conventions-vigilance.md § Artifact Evolution
  • Operational mandates: src/apothem/rules/operational-mandates.md CM-1–10
  • Plan suite reference: src/apothem/skills/plan-suite/master-template.md
  • Release notes: CHANGELOG.md

On this page