Watchman docs

Install

The hosted installer selects the current platform archive, verifies its checksum and exact member set, checks GitHub artifact provenance when a compatible gh CLI is available, and installs the public binary as watchman. Linux packages are static musl executables with no build-runner GLIBC requirement.

shell · install release

curl -fsSL https://tools.yshubham.com/watchman/install.sh | sh
watchman version

Install the agent skill

The skill gives compatible coding agents the full command map, safety boundary, report contracts, and operational recipes.

shell · all supported agents

npx skills add bas3line/rool-repo \
  --skill watchman --agent '*' --global --yes

agent-readable source

Agents can fetch this guide without parsing HTML: /docs/watchman/reference.md.

Agent protocol

Run watchman version and watchman --help.
Require version 0.8.3 or newer for this reference; stop and upgrade an older binary.
Run watchman <command> --help before composing flags. Never invent an option.
Choose the narrowest workflow that answers the operator's request.
Use JSON for one automation artifact and NDJSON for streams, history, or CI.
Return the redacted command, UTC window, scope, sources, typed status, exit code, failed gates, and nonclaims.

Choose a workflow

Need	Command	Traffic
Point-in-time node report	`snapshot`	passive
Live terminal control room	`top` / `watch`	passive
Prometheus + report API	`serve` / `exporter`	passive
GPU process ownership	`ps`	none
Driver and telemetry checks	`doctor`	passive
Local server fingerprint	`runtime inspect`	none
Safetensors validation	`artifact inspect`	none
Weights + KV planning	`capacity`	none
Small inference SLO probe	`canary`	active
Concurrency ladder	`benchmark saturation`	active
Saved benchmark gate	`benchmark compare`	none
Saved canary gate	`rollout`	none
Historical evidence	`history` / `compare`	none

Node triage

Start with a point-in-time report, process attribution, and a source diagnostic. These commands collect evidence without sending completion requests.

shell · incident baseline

watchman snapshot --all --details --format json > snapshot.json
watchman ps --format json > processes.json
watchman doctor --format json > doctor.json

Add --probe http://127.0.0.1:8000 to doctor when a loopback inference runtime exposes supported Prometheus metrics. Use watchman top --watch 2s only for an interactive view; prefer snapshot JSON for automation and handoff.

Runtime, artifacts + capacity

Fingerprint explicit PIDs

shell · privacy-safe local evidence

watchman runtime inspect \
  --pid 4242 --pid 4243 \
  --format json

An incomplete runtime fingerprint exits 2 by default. --allow-incomplete accepts best-effort evidence; it does not make the evidence complete.

Inspect metadata and plan memory

shell · no tensor payload load

watchman artifact inspect \
  /models/served-model/model.safetensors.index.json \
  --format json

watchman capacity \
  --model-config /models/served-model/config.json \
  --artifact /models/served-model/model.safetensors.index.json \
  --gpu-vram 24 --gpus 1 --tp 1 --weight-bits 16 \
  --format json

planning evidence

A capacity result does not prove runtime placement, loaded residency, allocator behavior, or that the model will start. Validate it on the exact serving topology.

Active inference tests

authorization boundary

canary and benchmark saturation send real chat-completion requests. Confirm the endpoint, synthetic workload, concurrency, token budget, and time window before running either command.

Small canary

shell · correctness + latency SLO

watchman canary \
  --base-url http://127.0.0.1:8000/v1 \
  --model served-model \
  --count 10 --concurrency 2 \
  --max-ttft 2s --max-e2e 10s \
  --min-success-percent 100 \
  --format json > canary.json

Bounded saturation ladder

shell · closed-loop exact stages

watchman benchmark saturation \
  --base-url http://127.0.0.1:8000/v1 \
  --model served-model \
  --concurrency-stages 1,2,4,8 \
  --warmup-requests-per-worker 2 \
  --requests-per-worker 20 \
  --verify-concurrency 8 \
  --max-error-percent 1 \
  --format json > benchmark.json

This is a single-generator, closed-loop test. It can hide coordinated omission; the client or network can bottleneck first; repeated prompts may hit prefix caching; concurrency is not GPU occupancy. A highest accepted tested point is not production capacity.

Offline deployment gates

Compare like-for-like artifacts without sending traffic to either endpoint.

shell · saturation regression

watchman benchmark compare baseline.json candidate.json \
  --max-p95-ttft-regression-percent 10 \
  --max-p95-e2e-regression-percent 10 \
  --min-successful-rps-ratio 0.95 \
  --min-completion-token-goodput-ratio 0.95 \
  --max-error-percent-increase 1 \
  --fail-on-regression --format json

shell · canary rollout

watchman rollout baseline-canary.json candidate-canary.json \
  --max-p95-ttft-regression-percent 10 \
  --max-p95-e2e-regression-percent 10 \
  --fail-on-regression --format json

Do not compare different workload identities, plans, policies, or exact stage ladders as though they were equivalent. Preserve incompatible and not_evaluable results.

Monitoring service

Keep the listener on loopback and use a private token file. The service exposes /livez, /metrics, /healthz, and /api/v1/report.

shell · authenticated loopback service

watchman serve \
  --interval 5s \
  --listen 127.0.0.1:9400 \
  --api-token-file /run/secrets/watchman-api-token \
  --history /var/lib/watchman/history.ndjson

Put remote access behind an authenticated, rate-limited TLS proxy. Do not use --no-api-auth outside an explicitly accepted loopback debugging run.

Evidence contract

Exit	Meaning
`0`	The workflow completed and every selected policy passed.
`1`	Usage, setup, validation, collection, I/O, or encoding prevented a valid workflow.
`2`	Completed fail-closed evidence: a selected policy failed, evidence was incomplete, capacity did not fit, or an enforced comparison regressed or was not evaluable. Inspect the emitted report.

Hardware reports use schema_version: 3. Runtime fingerprints use version 1, canaries version 2, saturation benchmarks version 1, and rollout results version 1.

do not flatten evidence

Missing samples, unavailable telemetry, incomplete usage, incompatible identity, a zero baseline, and not_evaluable evidence are not passes.

Safety boundary

Passive and offline workflows do not authorize active inference traffic.
Use synthetic, non-sensitive prompts. Treat reports, histories, canaries and bundles as private operational artifacts.
Prefer file-backed credentials. Never expose API keys, tokens, prompt content, generated output, URL queries, or raw environment values.
Prefer HTTPS for non-loopback targets. Cleartext remote HTTP requires explicit acceptance.
Watchman never changes clocks, power limits, compute modes, MIG state, or workloads.
A result proves only what its recorded plan, policy, sample set, and source evidence support—not causality, maximum production capacity, cost, significance, or an SLA.

Full references

CLI referenceEvery command, flag, limit, environment variable, output mode, and exit code. Command recipesFocused operating patterns for common inference workflows. ProfilesVersioned configuration, precedence, permissions, and secrets. Report schemasMachine contracts and the exact limits of each evidence type. Rollout gatesCanary identity checks, comparison formulas, and CI behavior. Benchmark gatesExact-stage performance comparisons and fail-closed outcomes.