<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <id>https://agenticdevops.github.io/aoh/field-notes</id>
    <title>AOH Field Notes</title>
    <updated>2026-07-17T00:00:00.000Z</updated>
    <generator>https://github.com/jpmonette/feed</generator>
    <link rel="alternate" href="https://agenticdevops.github.io/aoh/field-notes"/>
    <subtitle>AOH — Agentic Ops Harness Blog</subtitle>
    <icon>https://agenticdevops.github.io/aoh/img/favicon.ico</icon>
    <entry>
        <title type="html"><![CDATA[Inventory — the Ansible idea AOH needed]]></title>
        <id>https://agenticdevops.github.io/aoh/field-notes/2026/07/17/fleet-inventory</id>
        <link href="https://agenticdevops.github.io/aoh/field-notes/2026/07/17/fleet-inventory"/>
        <updated>2026-07-17T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[The pack/binding split has always leaned on an Ansible comparison — packs are]]></summary>
        <content type="html"><![CDATA[<p>The pack/binding split has always leaned on an Ansible comparison — packs are
roles, bindings are inventory entries. What I hadn't built yet was the actual
inventory file. Every binding install was still one binding, one command, one
workspace. This week that changed: <code>site.yaml</code> plus a <code>bindingsDir</code> of binding
files, and <code>aoh install --site</code> fans out across all of them in one shot.</p>
<p>The part I spent the most time on wasn't the fan-out loop itself — that's a
straightforward per-binding install with failure isolation, so one bad binding
doesn't take the rest of the fleet down. It was <code>site.lock.yaml</code>. A site's pack
source points at a git ref like <code>main</code>, and <code>main</code> moves. Once a site fans out to
a dozen workspaces, "what commit is actually running" stops being a question you
can answer by eyeballing a YAML file — you need a pin, and you need the install
path to refuse to guess.</p>
<p>So <code>aoh install --site</code> won't run without a lock that agrees with <code>site.yaml</code>. It
resolves every pack through the lock's <code>resolvedCommit</code>, never by re-resolving the
ref itself. <code>aoh lock</code> only writes entries that don't exist yet; <code>aoh lock --update</code> is the only thing that moves a pin forward, and it demands <code>--yes</code> if
the <em>source</em> changed, not just the commit. I proved this end to end with a real
git fixture: lock a pack, push a new commit upstream, re-install without
updating — the old commit is still what lands. Only after <code>aoh lock --update</code> does
the new one show up, and a commit that deletes a file actually removes it from
every workspace, not just adds the new ones.</p>
<p>That last part — convergence, not just fan-out — is what makes this feel like
inventory rather than a batch script. Ansible's inventory never made you think
about drift because the module system handled convergence per-host. AOH's install
path now does the same per-workspace, and the lock is what makes "per-workspace"
mean something across an entire fleet instead of one binding at a time.</p>]]></content>
        <author>
            <name>Gourav Shah</name>
            <uri>https://agenticops.tv</uri>
        </author>
        <category label="fleet" term="fleet"/>
        <category label="ansible" term="ansible"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Building AOH's docs the way AOH builds packs]]></title>
        <id>https://agenticdevops.github.io/aoh/field-notes/2026/07/16/building-aoh-docs-like-aoh-packs</id>
        <link href="https://agenticdevops.github.io/aoh/field-notes/2026/07/16/building-aoh-docs-like-aoh-packs"/>
        <updated>2026-07-16T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Last one in this run of Field Notes (see #1 for where this all started) — and it's about the site you're reading right now.]]></summary>
        <content type="html"><![CDATA[<p>Last one in this run of Field Notes (see <a href="https://agenticops.tv/" target="_blank" rel="noopener noreferrer" class="">#1</a> for where this all started) — and it's about the site you're reading right now.</p>
<p>I didn't want the docs to be a bolt-on written in a single pass at the end. AOH's whole pitch is that good agentic work follows a loop — brainstorm the shape, write the plan down, execute in reviewable slices, check the result against reality — so I built the docs site the same way I build a pack.</p>
<p>It started as a brainstorm, not a docs outline: what does someone actually need in the first sixty seconds, what's a concept versus a tutorial versus a reference page, which claims in the existing markdown are real versus roadmap. That turned into a written plan — sections, deck filenames, which pages get interactive components, a hard build gate (<code>onBrokenLinks: 'throw'</code>, zero tolerance) — before any content got written.</p>
<p>Then it went to subagent-driven execution: one task per slice (scaffold, concepts, decks, getting-started, tutorials, reference, this landing-and-blog task), each one reviewed on its own before the next started. A concept page couldn't claim a feature that wasn't in the code — the drift model, the eval runner, non-Hermes adapters are all marked "planned" rather than described as if they were shipped. And the whole thing closes with a full-branch review before it's called done, the same review discipline that gates a pack going into <code>collections/core/</code>.</p>
<p>It's a small thing, maybe, but it's the same conviction the harness itself is built on: process worth having for code is process worth having for the words describing that code, too.</p>
<p>If you want to see where it landed, the repo's right there — <a href="https://github.com/agenticdevops/aoh" target="_blank" rel="noopener noreferrer" class="">github.com/agenticdevops/aoh</a>. Try it: clone it, run <code>uv run aoh validate</code> against one of the example packs, and see if the docs actually got you there without detours.</p>]]></content>
        <author>
            <name>Gourav Shah</name>
            <uri>https://agenticops.tv</uri>
        </author>
        <category label="meta" term="meta"/>
        <category label="docs" term="docs"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[The Claude Code permission wall most people miss]]></title>
        <id>https://agenticdevops.github.io/aoh/field-notes/2026/07/16/two-walls</id>
        <link href="https://agenticdevops.github.io/aoh/field-notes/2026/07/16/two-walls"/>
        <updated>2026-07-16T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Continuing the build log — Field Note #2 proved the RBAC wall was real. This one's about a second wall that shows up only in Claude Code, and why I still don't trust it alone.]]></summary>
        <content type="html"><![CDATA[<p>Continuing the build log — <a class="" href="https://agenticdevops.github.io/aoh/docs/tutorials/kubeops-readonly">Field Note #2</a> proved the RBAC wall was real. This one's about a second wall that shows up only in Claude Code, and why I still don't trust it alone.</p>
<p>Shipping the Claude Code and Codex adapters this week meant answering a question I'd put off: does either runtime have a config surface that can say "no" to <code>kubectl delete</code> before the request leaves the agent? Codex, no — <code>execpolicy</code> rules match a literal command prefix, and three trivial forms slip past it: put <code>--context</code> first, use an absolute binary path, or wrap the call in <code>sh -c</code>. All three return no decision at all, silently.</p>
<p>Claude Code surprised me. <code>.claude/settings.json</code> supports <code>permissions.deny</code> at the subcommand level — name <code>kubectl delete</code>, <code>kubectl apply</code>, <code>helm upgrade</code> specifically, and Claude Code refuses the tool call outright. Most people evaluating it for ops work don't know this exists; I didn't, until I went looking while building this adapter. The AOH-generated workspace now sets it for every mutation verb in the shared kubectl/helm list.</p>
<p>A literal deny list has the same blind spot Codex's rules do, so I paired it with a <code>PreToolUse</code> hook — <code>kubectl-guard.sh</code> — that actually parses the command instead of pattern-matching it: strips <code>sudo</code>/<code>env</code> wrappers, unwraps <code>sh -c</code>, strips absolute paths, tokenizes past flags for the real verb. Fail-closed everywhere: bad JSON, missing <code>jq</code>, any shell metacharacter, all block. It catches all three forms that beat Codex's rules. It also has a sharp edge I didn't expect until testing it: it blocks <em>every</em> compound Bash command, kubectl or not, because it can't safely tokenize syntax it doesn't parse. <code>kubectl get pods | grep Pending</code> gets blocked right alongside <code>kubectl delete pod x</code>. Documented, not silently patched around.</p>
<p>Here's the honest part. Neither the deny list nor the hook is a boundary — they're guardrails, best-effort, scoped to commands routed through Claude Code's own Bash tool, blind to anything outside that session. The actual boundary is what it was in the Hermes tutorial: a scoped ServiceAccount whose RBAC the API server checks on every request, regardless of what Claude Code's config catches. If the hook has a bug, that's a worse experience, not a security failure — the cluster still says no.</p>
<p>Try it: <a class="" href="https://agenticdevops.github.io/aoh/docs/tutorials/kubeops-claude-code">the new Claude Code tutorial</a> walks through generating the workspace, reading the deny list and hook yourself, and watching both walls hold — one best-effort, one absolute.</p>]]></content>
        <author>
            <name>Gourav Shah</name>
            <uri>https://agenticops.tv</uri>
        </author>
        <category label="kubernetes" term="kubernetes"/>
        <category label="safety" term="safety"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[A Kubernetes agent that can't kubectl delete]]></title>
        <id>https://agenticdevops.github.io/aoh/field-notes/2026/07/15/a-read-only-kubernetes-agent</id>
        <link href="https://agenticdevops.github.io/aoh/field-notes/2026/07/15/a-read-only-kubernetes-agent"/>
        <updated>2026-07-15T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[More from the build log — Field Note #1 talked about the shape of the harness; this one's about proving a guardrail is real instead of just asserting it.]]></summary>
        <content type="html"><![CDATA[<p>More from the build log — <a href="https://agenticops.tv/" target="_blank" rel="noopener noreferrer" class="">Field Note #1</a> talked about the shape of the harness; this one's about proving a guardrail is real instead of just asserting it.</p>
<p>I wanted to answer one question honestly: if I tell an AOH pack its agent is <code>kubectl-readonly</code>, is that a comment or is it true? "Instructed not to delete things" and "physically incapable of deleting things" are very different claims, and I only wanted to ship the second one.</p>
<p>The answer ended up living outside the agent entirely. <code>install-hermes-agent --binding</code> now generates a <code>provision.sh</code> that a human runs once, by hand — AOH never touches the cluster itself, it only writes files. That script creates a dedicated <code>ServiceAccount</code>, a <code>ClusterRole</code> scoped to <code>get</code>/<code>list</code>/<code>watch</code>, binds them together, and writes a scoped <code>kubeconfig</code>. The agent launches with that kubeconfig, not yours.</p>
<p>Then I proved it without the agent anywhere in the loop. Same kubeconfig, two commands, on a real <code>kind</code> cluster:</p>
<div class="language-text codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#393A34;--prism-background-color:#f6f8fa"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="color:#393A34;background-color:#f6f8fa"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#393A34"><span class="token plain">kubectl --kubeconfig "$KC" get pods -A                                        # works</span><br></div><div class="token-line" style="color:#393A34"><span class="token plain">kubectl --kubeconfig "$KC" delete pod coredns-668d6bf9bc-kw88m -n kube-system  # Forbidden</span><br></div></code></pre></div></div>
<p>The API server itself rejected the delete — <code>Error from server (Forbidden): ... cannot delete resource "pods"</code>. Nothing about how the request was made changed between the two calls; only the RBAC attached to the identity that sent it. That's the guardrail I wanted: it lives in the target platform, not in a prompt telling the agent to behave.</p>
<p>While that provision-script generator was under final review, a real bug surfaced: binding values (context names, namespaces) were being interpolated straight into bash without validation — a textbook shell-injection opening if a binding file ever came from somewhere less trusted than my own repo. The fix was a strict allowlist regex on every value before it touches the generated script, with a <code>PackError</code> if anything outside <code>[A-Za-z0-9._-]</code> shows up. Caught before it shipped, but a good reminder that "generates a shell script" is a security-sensitive code path, review or no review.</p>
<p>Full walkthrough, copy-pasteable commands, and the honesty note about what "read-only" doesn't cover: <a class="" href="https://agenticdevops.github.io/aoh/docs/tutorials/kubeops-readonly">the kubeops-readonly tutorial</a>. If you've got a <code>kind</code> cluster lying around, try it — watch the API server say no.</p>]]></content>
        <author>
            <name>Gourav Shah</name>
            <uri>https://agenticops.tv</uri>
        </author>
        <category label="kubernetes" term="kubernetes"/>
        <category label="safety" term="safety"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Killing the Workflow kind]]></title>
        <id>https://agenticdevops.github.io/aoh/field-notes/2026/07/14/killing-the-workflow-kind</id>
        <link href="https://agenticdevops.github.io/aoh/field-notes/2026/07/14/killing-the-workflow-kind"/>
        <updated>2026-07-14T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Continuing the build-in-public thread from Field Note #1 — this one's about deleting a kind I'd only just shipped.]]></summary>
        <content type="html"><![CDATA[<p>Continuing the build-in-public thread from <a href="https://agenticops.tv/" target="_blank" rel="noopener noreferrer" class="">Field Note #1</a> — this one's about deleting a kind I'd only just shipped.</p>
<p>AOH's spec started with a <code>Workflow</code> kind: a YAML file that named a sequence of skills a role should run in order. It felt obvious at the time — Ansible has playbooks, so AOH should have something that chains steps. I wrote it, wired the validator to it, had the Hermes adapter emit commands for it.</p>
<p>Then I actually looked at what was in one. A workflow file was a name, a list of skill references, and maybe a note on ordering. Every field in it either duplicated something already declared on the role, or restated the skill's own name back at me. It carried zero logic of its own — it was a reference bundle wearing a kind's clothing.</p>
<p>Around the same time I was reading through how Superpowers structures its own skills, and the pattern clicked: a multi-step process isn't a different <em>kind</em> of thing from a skill — it's a skill whose body happens to orchestrate other skills by name. <code>platform-sre-triage</code> doesn't need a <code>Workflow</code> wrapper; it needs a <code>SKILL.md</code> that says "run X, then branch on the result, then run Y." That's still a skill. The runtime doesn't care that it's "a workflow" — it just executes what the <code>SKILL.md</code> tells it to.</p>
<p>So <code>Workflow</code> came out. The multi-skill workflows in the example packs became process skills — same content, one fewer kind to validate, reference, and keep in sync with the role. While I was in there breaking things, I also renamed <code>agents/</code> to <code>roles/</code> (role is the real-world word; "agent" was doing double duty with the runtime concept) and picked <code>ops:</code> as the canonical command prefix Hermes emits, instead of <code>aoh:</code>.</p>
<p>Fewer kinds, same expressiveness, less to keep consistent. If you want the current shape of the spec — what's mandatory, what's opt-in, and where process skills fit — the <a class="" href="https://agenticdevops.github.io/aoh/docs/reference/pack-spec">pack spec reference</a> has the whole thing. Try it: run <code>aoh validate</code> against a skills-only pack and watch how little ceremony it actually takes to get to a valid one.</p>]]></content>
        <author>
            <name>Gourav Shah</name>
            <uri>https://agenticops.tv</uri>
        </author>
        <category label="spec" term="spec"/>
        <category label="design" term="design"/>
    </entry>
</feed>