Capability Smoke Test
A single synthetic workflow package, built from scratch, whose only job is to exercise every kind of thing a Chanakya workflow step can do — reasoning, tools, the sandbox, engines, human gates, and every deliverable shape — against small mock data instead of a real engagement.
Capability inventory
Every ability a package.yaml step can reach for, grouped the
way the platform's own docs group them. “Proves” names the step in Part 02
that exercises it against mock data; documented only means it's real and worth
knowing about, but deliberately not wired into this package's golden path — reasons are
in Part 04.
Reasoning & document tools
| Capability | Mechanism | Where |
|---|---|---|
| Pure reasoning, no tools at all | tools: [] | scope_confirm_gate |
| Read one document / list the workspace | read_document · list_documents · search_documents | read_documents |
| Bulk-read every document, no raw text in context | for_each(action="read", items="workspace") | read_documents |
| Cross-document consistency check | compare_documents | read_documents |
| Note the notebook: search / read / write a scratchpad note | search_scratchpad · get_note · write_note | throughout |
| Exact arithmetic (never eyeballed) | compute | tie_out_check, ledger_bulk_scan |
| Self-check a step's own output shape | check_output_schema | executive_summary |
| Bulk judgement, one fresh sub-agent per item | for_each(action="answer") | bank_answer_review |
| Discover what's callable by name | list_skills · list_engines · describe_engine | ledger_bulk_scan, sandbox steps |
Sandbox & compute
| Capability | Mechanism | Where |
|---|---|---|
| Pinned skill, inputs already known — no container, no charge | run_analysis(skill=, inputs=) | tie_out_check |
| Pinned skill, sandbox works out the inputs first | run_analysis(skill=, instructions=, document_ids=) | sandbox_marshal_check |
| Free Python written & run on the spot — nothing audited | run_analysis(instructions=) | sandbox_code_check |
| Declarative single engine call, platform-resolved inputs | engine_bindings (mode: direct) | ledger_bulk_scan |
| Declarative one-call-per-row engine call | engine_bindings (mode: for_each_rowset) | rowset_bulk_scan |
| Domain-value → engine-literal translation | engine_bindings.enum_map | rowset_bulk_scan |
| Structured ingestion of an uploaded sheet | ingest_rows → set_column_mapping | rowset_bulk_scan |
| Bulk table addressed by handle, never inlined | list_tables → describe_table → {"$table": …, "map": …} | ledger_bulk_scan |
| Refuse a table whose printed total doesn't foot | footing gate → TABLE_UNREADABLE | documented only |
| Refuse to compute over an empty population | required_population_fields → ENGINE_INPUT_STARVED | documented only |
Human-in-the-loop
| Capability | Mechanism | Where |
|---|---|---|
| Zero-agent pure gate (no model turn at all) | gate_only: true | intake_gate, threshold_gate |
| File upload with a stated cost-of-skipping | kind: file_upload + accept + impact_note | intake_gate |
| Bounded, unit-labelled, order-constrained numbers | min · max · unit · less_than_field | threshold_gate |
| Attestation vs. an ordinary yes/no fact | must_be_true vs. plain bool | intake_gate, sign_off_fill |
| Why a value is being shown at all | provenance: judgment | threshold_gate |
| Repeating structured input | kind: array_of_objects + item_fields | sign_off_fill |
| Reusable, already-registered pause + verify pair | pre_pause_handler / verifier_handler = workspace_resource_picker_* | scope_confirm_gate |
| Capped retry on a rejected submission | max_attempts | scope_confirm_gate |
| Approve / correct / deterministically recompute | review.mode: approve_edit_rerun + recompute_on_edit | tie_out_check |
| A different actor must submit this gate | sod_against_step | sign_off_fill |
| Skip only the pause, not the step that fed it | human_input_after.skip_when | bank_answer_review |
| Skip the entire step (no agent run, no cost) | step-level skip_when | rowset_bulk_scan |
| Agent asks for a value it needs, mid-step | request_human_input | resilience_checks |
| Agent stops and names the external fix needed | abort_step | resilience_checks |
| Multi-actor chained approval | N sequential human_input_after steps, later SoD'd against earlier | threshold_gate → … → sign_off_fill |
| OTP-style expiring gate | expires_in_seconds + an OTP relay handler | documented only |
The notes contract
| Capability | Mechanism | Where |
|---|---|---|
| Declared, enforced hand-off between steps | produces_notes / consumes_notes | tie_out_check → sign_off_fill, … |
| Note auto-derived from structured output — no dual-write | produces_notes: [{label, from_output: true, include}] | ledger_bulk_scan |
| A note that correctly may not be written this run | may_produce_notes | resilience_checks |
| A missing upstream note fails fast, not slow | consumes_notes entry gate | sign_off_fill |
| A completing step whose containers are vacuously empty | vacuous hand-off tripwire | documented only |
Deliverables
| Capability | Mechanism | Where |
|---|---|---|
| Bound xlsx fill (Phase A/B → WorkbookWriter) | output_template_id (xlsx) | sign_off_fill |
| Bound docx fill from cited notes | cite_note_for_fill → edit_docx | docx_template_fill* |
| Nested-table cell inside a docx | docx_nested_table_cell | docx_template_fill* |
| A schedule that grows to fit the rows, never a fixed grid | annexes + fill_map + empty_reason | sign_off_fill |
| Deterministic producer→field bridge, no LLM guess | fill_map | sign_off_fill |
| A cell the fill model is never shown | signoff: true | sign_off_fill |
| A conclusion only the model can author | judgment: true | sign_off_fill |
| A slot that may legitimately stay blank | optional: true | sign_off_fill |
| A written cell re-checked against its engine | engine_check | sign_off_fill |
| A workbook with no authored binary at all | generated: true | Summary Workpaper template |
| Ad-hoc Word authoring from nothing | docx toolkit (create/inspect/replace/write_table/style/save) | freeform_docx_memo |
| Free-form prose deliverable | emit_artifact(output_format="markdown") | executive_summary |
| Why a step can't emit_artifact an xlsx/docx | FORMAT_NOT_ALLOWED_IN_WORKFLOW boundary | documented only |
| Cite + write collapsed into one call | fill_from_notes (fill_template:) | documented only |
* docx_template_fill is the one step gated on the templates decision in Part 03.
Lifecycle & governance
| Capability | Mechanism | Where |
|---|---|---|
| Author → validate → publish → provision → apply | chanakya-pkg | Part 05 |
| End-to-end fill dry run before publish | validate --fill-check + sample_values.json | Part 05 |
| Runaway guards: no-progress, read-churn, tool-call cap, wall clock, per-item cap | step_executor reliability backstops | documented only |
The package blueprint
One package, four mock fixtures, two purpose-built skills, fifteen steps. Nothing here
touches real CA math — capability_math_check and
capability_row_scan are intentionally generic so the package
tests the platform, not an engagement.
Package identity
- apiVersion
- chanakya.dev/package/v1
- metadata.slug
- capability-smoke-test
- metadata.category
- platform-diagnostic
- required_parameters
- [financial_year]
Mock fixtures
| File | Shape | Feeds |
|---|---|---|
| mock_general_ledger.xlsx | ~32 rows — Date, Particulars, Amount + a printed total row that foots correctly | the table corpus ($table path) |
| mock_vendor_invoice.txt | one invoice — seller/buyer, 4–5 line items, subtotal, GST, stated total | read_documents, tie_out_check |
| mock_bank_statement.txt | ~15 transaction lines, opening/closing balance, a few large/cash entries | read_documents, sandbox_marshal_check, bank_answer_review |
| mock_disbursements.csv | ~12 rows — Date, Narration, Amount, Mode (NEFT/UPI/Cash/Journal) | the structured-ingestion path (ingest_rows) |
Every name in the fixtures — entities, payees, amounts — is an invented placeholder; none resembles a real filing.
Two skills, the two canonical shapes
capability_math_check
Scalars in, scalars out — mirrors skills/materiality/. Inputs: book_total, external_total, tolerance_pct (all Decimal). Output: difference, difference_pct, tie_out_status. No required_population_fields — a scalar tie-out has no population to starve.
capability_row_scan
A table in, an exceptions table out — mirrors skills/ledger_exception_scan/. Inputs: fy, threshold, rows: list[LedgerRow] (date, particulars, amount, channel?). Output: total_amount, row_count, exceptions, exceptions_count. Declares required_population_fields: [rows] and a full presentation: block per output field.
Step flow
Step table
| # | Step | Tags | Key tools | Proves |
|---|---|---|---|---|
| 1 | intake_gateUpload the ledger, invoice, statement; attest this is synthetic | gate | — | gate_only, file_upload+accept+impact_note, must_be_true attestation |
| 2 | scope_confirm_gateRestate scope; pick the primary invoice via the resource picker | agentgate | tools: [] | reasoning-only step; reusable pre_pause_handler / verifier_handler; max_attempts |
| 3 | threshold_gateCA sets the major / minor exception thresholds and tolerance | gate | — | gate_only, min/max/unit, less_than_field, provenance: judgment |
| 4 | read_documentsFan the invoice and statement into open notes | agent | for_each, compare_documents | bulk read fan-out; cross-document consistency |
| 5 | tie_out_checkTie invoice line items to its stated total | agentsandboxgate | run_analysis, compute | run_analysis mode 1; review + recompute_on_edit |
| 6 | sandbox_code_checkFree Python over the ledger table — largest / average transaction | agentsandbox | run_analysis, list_tables | run_analysis mode 3 — instructions only, nothing audited |
| 7 | sandbox_marshal_checkMarshal the statement's open/close figures into the scalar skill | agentsandbox | run_analysis, list_documents | run_analysis mode 2 — coding agent marshals, pinned skill computes |
| 8 | ledger_bulk_scanScan the ledger for exceptions via the table corpus | agentengine | list_tables, describe_table, capability_row_scan | $table handle; engine_bindings direct + mandatory + $param/$step constants + outputs; G5 auto-note |
| 9 | rowset_bulk_scanScan the disbursements sheet, one engine call per row | agentengine | ingest_rows, set_column_mapping, for_each | structured ingestion trio; engine_bindings for_each_rowset + enum_map; step-level skip_when |
| 10 | bank_answer_reviewFresh sub-agent judgement on each flagged transaction | agentgate | for_each(answer) | bulk judgement fan-out; human_input_after.skip_when (pause only, step still runs) |
| 11 | resilience_checksConfirm preconditions; the two exit doors, wired but untriggered | agent | request_human_input, abort_step | dynamic data ask vs. dynamic external-action abort; may_produce_notes |
| 12 | freeform_docx_memoBuild a Word memo from nothing | agent | docx (create/write_table/style/save) | ad-hoc docx authoring — no template, no field_schema |
| 13 | sign_off_fillFill and sign the xlsx working paper | agentgate | cite_note_for_fill, compute | output_template_id; fill_map; annex+empty_reason; signoff withheld; SoD; array_of_objects |
| 14 | docx_template_fillFill the nested-table verification note | agent | cite_note_for_fill, edit_docx | bound docx fill; docx_nested_table_cell — conditional, see Part 03 |
| 15 | executive_summarySelf-check the run, then emit a markdown summary | agent | check_output_schema, emit_artifact | free-form prose deliverable; the xlsx/docx-from-a-step boundary |
Three steps, written out
The richest three, verbatim enough to build from. The other twelve follow the same conventions.
- id: tie_out_check
name: Tie the invoice line items to its stated total
tools: [run_analysis, list_skills, search_scratchpad, get_note, write_note, compute]
required_inputs:
- from_step: threshold_gate
instructions:
- Find the primary vendor invoice note written by read_documents. Sum its
line items with compute — never by eye.
- Call run_analysis(skill="capability_math_check", inputs={book_total: the
summed line items, external_total: the invoice's own stated total,
tolerance_pct: threshold_gate's tolerance_pct}). You already know both
figures, so pass them directly — do not describe them in prose.
- Write ONE cited note labelled "tie_out" holding the difference,
difference_pct, tie_out_status and the engine's audit_id.
output_schema:
book_total: number
external_total: number
tolerance_pct: number
difference: number
difference_pct: number
tie_out_status: string
tie_out_audit_id: string
required_outputs: [book_total, external_total, tolerance_pct, difference, tie_out_status]
produces_notes: [tie_out]
human_input_after:
title: Confirm the invoice tie-out
fields:
- { name: confirm, kind: bool, required: true }
review:
mode: approve_edit_rerun
max_refine_rounds: 2
editable_fields: [book_total, external_total, tolerance_pct]
recompute_on_edit:
engine: capability_math_check
audit_key: tie_out_audit_id
input_map:
book_total: book_total
external_total: external_total
tolerance_pct: tolerance_pct
output_map:
difference: difference
difference_pct: difference_pct
tie_out_status: tie_out_status
trigger_fields: [book_total, external_total, tolerance_pct]- id: ledger_bulk_scan
name: Scan the general ledger for exceptions
tools: [list_tables, describe_table, capability_row_scan, list_engines,
describe_engine, search_scratchpad]
required_inputs:
- from_step: threshold_gate
instructions:
- Call list_tables, then describe_table on the general-ledger table to
confirm its columns and control total.
- Call capability_row_scan directly. Its inputs are bound below — pass the
{"$table": "the ledger table", "map": {date: Date, particulars:
Particulars, amount: Amount}} handle as rows and nothing else; the
platform supplies fy and threshold.
engine_bindings:
- engine: capability_row_scan
mode: direct
mandatory: true
constants:
fy: "$param.financial_year"
threshold: "$step.threshold_gate.major_threshold"
outputs:
total_amount: total_amount
exceptions_count: exceptions_count
exceptions: exceptions
output_schema:
total_amount: number
exceptions_count: number
exceptions: list
required_outputs: [total_amount, exceptions_count, exceptions]
produces_notes:
- label: ledger_scan
from_output: true
include: [total_amount, exceptions_count, exceptions]- id: rowset_bulk_scan
name: Scan the disbursements sheet for exceptions
skip_when: "intake_gate.bulk_data_path:table_corpus"
tools: [list_documents, ingest_rows, set_column_mapping, for_each,
search_scratchpad, write_note, compute]
required_inputs:
- from_step: threshold_gate
instructions:
- Call ingest_rows on the disbursements sheet, then set_column_mapping
confirming the sheet's own column names.
- Call for_each(action="compute", engine="capability_row_scan",
rowset_id=the ingested rowset) — the mapping below builds every row's
payload; do not re-type rows yourself.
- After it returns, total the exceptions across the per-row notes with
compute, then write_note(label="rowset_scan_summary", ...) ONE
consolidated note.
engine_bindings:
- engine: capability_row_scan
mode: for_each_rowset
input_map:
date: Date
particulars: Narration
amount: Amount
enum_map:
field: channel
from_field: Mode
values: { NEFT: bank, UPI: bank, Cash: cash, Journal: adjustment }
constants:
fy: "$param.financial_year"
threshold: "$step.threshold_gate.minor_threshold"
output_schema:
exceptions_count: number
total_amount: number
produces_notes: [rowset_scan_summary]Output templates
Two deliverables. One is settled; the other is the first open decision.
Capability Smoke Test — Summary Workpaper xlsx
generated: true — schema-only, no binary to author. Carries: a bridged header, a required: true tie-out cell, a judgment: true overall-assessment cell, an optional: true follow-up cell, a signoff: true block (prepared/reviewed, names + dates), an annexes exceptions schedule with empty_reason and a SUM formula, and one engine_check bound to a genuinely-computed field.
Capability Smoke Test — Verification Note docx
Hand-authored — a plain table of engagement facts plus one nested table (a document register with a per-row detail grid), to exercise docx_nested_table_cell. Needs a real .docx binary and one LLM add-output detection pass.
TOGETHER_API_KEY),
and a review of what it detected. Skipping it drops docx_template_fill
and docx_nested_table_cell coverage from the golden path — everything
else in Part 01 still runs, including the free-form docx authoring in step 12.
Deliberately out of scope
Real, registered, and documented above — just not wired into a run that has to stay green on mock data.
| Capability | Why not here |
|---|---|
| rule_pack: (FY-versioned statute) | no statutory rate table applies to synthetic figures |
| WritePlan durable rows | needs real reconciliation/vouch rows to attach to — out of scope for mock data |
| ais_fetch, gst_fetch, verify_pan/gstin, mca_*, surepass_* | real vendor credentials, metered credits, and (gst/ais) a live OTP round-trip |
| indian_kanoon, cite_law, query_reference_data | a live legal index, or a knowledge-tier stub with no corpus yet (E5, deferred) |
| trigger_workflow, list_workflows | orchestration tools that spawn a second workflow — risks recursive spawning in a self-contained demo |
| expires_in_seconds + OTP relay | the only registered relay verifier is an intentional not-yet-implemented placeholder |
| fill_from_notes | the same scratchpad fill as step 14, one call instead of two — a third template would be redundant coverage |
| engine_bindings.select_by | needs a genuine multi-candidate decision; see materiality-determination's assess_materiality for the reference |
| Adversarial fixtures — mis-footed total, empty rows, a dropped note | kept off the golden path so the reference run stays green; each is one fixture edit away from TABLE_UNREADABLE / ENGINE_INPUT_STARVED / missing_upstream_note |
Build & verify plan
What actually building this involves, in order. Nothing here touches the shared dev/prod registry unless asked separately.
- Write the two skill bundles under
apps/api/skills/capability_math_check/and.../capability_row_scan/— skill.yaml, SKILL.md, compute.py, examples/, tests/. Git-tracked, reviewed like any other code.Runuv run python -m pytest tests/test_skills/ skills/ -q— the bundle lints cover both automatically. - Write the four mock fixtures + package.yaml under
apps/api/packages/capability-smoke-test/— gitignored authoring scratch, exactly like every other package. - Author the xlsx (generated: true — no binary step) and, if the templates decision picks full coverage, the docx via
add-output. - Commit a
sample_values.json, thenpython -m cli.chanakya_pkg validate capability-smoke-test --fill-check. - Local-only publish + provision into fake-gcs (mirrors
make seed's own path) — never against the realbeyond-fill-templatesbucket. - Trigger a real run against the local Docker stack (
make up) with the four mock files uploaded, and watch every one of the fifteen steps go green. - Stays local. No
publish/provisionagainst the shared dev or prod registry unless you ask for that separately — this is a diagnostic tool, not a CA deliverable.