Take the Keys Away From Your Agent
- Event
- The Innovation Lab
- Overview
A process that runs model-authored code should hold no credential worth stealing and have no route to anywhere it could send one. A ground-up treatment of TCP, HTTP, TLS, certificate authorities and proxies, then the architecture they compose into: a placeholder key in the runner, the real one in a separate container that injects it on the way out, and a network with no gateway at all. Five interactive demos, including an egress simulator you can unlock to watch every control fail.
- Technologies
- SecurityNetworkingTLSCertificate AuthoritiesDockerAgents
Take the Keys Away From Your Agent
An agent that writes code and runs it needs a credential and a network. The obvious way to give it both — one container, the key in an environment variable, an ordinary bridged network — works on the first day and keeps working, which is the problem. That process takes instructions partly from things it read: a web page, a file, the output of a tool. Untrusted input, a live credential, and a route to every host on the internet, in one place.
You cannot close this at the model layer, because that means asking the component whose behaviour is in question to enforce its own limits. The controls that hold are the ones it cannot address: it holds no credential worth stealing, and it has no route to anywhere it could send one.
Not because the model is malicious, but because its behaviour is partly a function of input you do not control. Assume it will eventually do the worst thing available — then ask what is available.
A separate container holds the real credential and adds it on the way out. The runner carries a placeholder. Requests still succeed; the key is never in reach.
Not a firewall rule that could be misordered — an absent route. The kernel has nowhere to send the packet, for any protocol.
A pattern, not a product tour. Nothing here depends on a particular cloud, orchestrator or provider — provider.example stands in for whichever API you actually call. The five panels below are interactive; they are the article.
What is actually on the wire
Three protocols do all the work, and they stack. Click each band to see what it is and what an observer at that depth can read — then hit TLS off and watch what changes.
TCPthe pipe
A reliable, ordered stream of bytes between a port here and a port there. That is the whole job. It guarantees delivery and order, or a visible failure — and it has no opinion whatsoever about what the bytes mean.
Ciphertext. The observer still learns the addresses, the ports, the size of every packet and its timing. Encryption hides contents. It never hides that you spoke.
tap on the wire
IP 10.0.4.7.51314 > 203.0.113.9.443: Flags [P.], length 307 0x0000: 1703 0301 2e9a 4fe2 710c bb38 d56a 9104 0x0010: c722 ef1b 883d 5ea0 76f4 12cd 398b e657 0x0020: 2a90 dd14 6fb3 41c8 057e 92ab 3fdf 5085 0x0030: 57ba 0a77 fa1c 0927 1dd0 8a6e 7764 e1af 0x0040: f47d b8fc 4229 0b50 2e97 c678 9e84 1494 0x0050: fe89 adf5 6cc6 b190 8265 aa4b 376f 8f24 0x0060: e15d f40f ca6d f1a9 2b45 6ec0 9267 a84a 0x0070: 9358 47c5 9c26 5d20 e8bf 92cd 3c93 411d 0x0080: ba7a f224 ecaf b2d6 a460 eb83 2196 6367 0x0090: 5016 86eb 9e1c dafb 784e 485d f251 22c1 0x00a0: e797 5be1 c715 8109 9b7d 1b3f 8d8b 36cc 0x00b0: ac97 4eaf 6f4c ec9f 5ad7 2970 ffbd 15a2 0x00c0: 4aae 80bd 5c8b 55ff 1546 0e70 8da9 2fad 0x00d0: 2856 1f12 7be4 3cb5 4d04 30bf 7f01 6e5e 0x00e0: 4023 a876 1c0b a081 2687 8f35 23e9 57ae 0x00f0: b427 b76e 0e26 c5d0 65bf c891 4629 bae7 0x0100: 32fe dcc0 3a50 ecca 8b60 d6ef e626 4b4e 0x0110: 937d 79a7 ae7a 72ed b08a c8ca 2b02 debb 0x0120: 1f76 8d6e 4ca1 a790 ff93 9cd0 ace5 74ab 0x0130: 607d 91
The HTTP panel is identical in both modes, because HTTP does not change when you wrap it. Only who can read it in transit changes. That is the fact everything below rests on: TLS makes a request private from the network, not from the endpoints. Whoever completes the handshake reads the plaintext — credential included — and can therefore also rewrite it.
Which sounds like a description of an attack. It is also the mechanism for keeping a secret out of a process you do not trust. Telling those apart is the next section.
Who decides what you believe
TLS proves the server’s identity with a certificate — a name and a public key, signed by somebody else. That signature is checked against a certificate authority, whose own certificate is checked against another, until the walk reaches a self-signed root and asks one final question: is this root in my trust store? Run the walk, then run it again with a root your own deployment installed.
Every https:// request you have ever made without thinking.
- issued by
- itself
- valid for
- 20 years
- comes from
- Shipped in your operating system's trust store. You did not choose it — your OS vendor did, years ago, on your behalf.
- issued by
- Example Root CA X1
- valid for
- 5 years
- comes from
- Sent alongside the leaf, to bridge the gap to a root.
- issued by
- Example Trust CA G3
- valid for
- 90 days
- comes from
- Presented by whoever answered on that address, during the handshake.
Both chains verify, and the client cannot tell them apart, because there is nothing to tell apart — the checks are identical and both pass honestly. What differs sits upstream of all the cryptography: somebody decided which roots this machine believes. When an attacker does it we call it a compromise; when you do it so a component you operate can read traffic from one you do not trust, it is architecture.
Which puts the real boundary somewhere unglamorous: the file permissions on the root’s private key. Publish the certificate where the runner can read it, and put the key on a volume the runner has no mount for — not a stricter file mode, a volume it does not have. When two containers run as the same user id, a mode bit is a note rather than a wall.
Standing in the middle, two ways
A proxy is any process that makes a request on someone else’s behalf. The distinction that matters is not forward versus reverse — it is whether the proxy terminates the TLS. A CONNECT tunnel opens a raw pipe and then relays bytes it cannot read; a terminating proxy answers the handshake itself and sees everything. Send one request through both.
One connection, straight through. The same amber session runs from the runner all the way to the destination — the relay is just pipe.
Two connections. The runner's amber session ends at the proxy; a separate teal one continues outward, carrying a credential the runner never held.
The instinct in most security writing is that a proxy reading your plaintext is bad. That instinct is calibrated for a threat model where the client is you and the middlebox belongs to someone else. Invert it. Here the client is a process running code a model wrote, and the middlebox is yours — so the tunnel is the dangerous option, because a blind relay can only carry an authenticated request if the client already holds the credential.
Note the dependency: terminating requires the runner to accept a certificate the proxy minted, which requires it to trust a root you installed, which requires that root’s key to be somewhere the runner cannot read. Remove any one and the mechanism either stops working or stops meaning anything.
Separation has to be structural
Inside one process, every boundary you think you have is a convention. process.env is a flat namespace shared by every line of code in the program, including the code the model wrote thirty seconds ago. “The agent shouldn’t read that variable” is a code review comment, not a control — a stack trace prints it, a subprocess inherits it, a logging library ships it somewhere on purpose.
Put the secret in a different process and none of that is available; put it in a different container and the filesystem and network namespace go too. Enforcement moves from your discipline to the kernel, which does not get tired and cannot be talked into things. Hence the single rule everything here reduces to: the credential and the untrusted code must never share a process. Click through the five and check the blast radius on each.
locked
No gateway. No route out, for any protocol.
the wall
Everything that bridges both sides. The whole list.
egress
Ordinary bridged networking. Reaches the internet.
runneruntrusted
The agent process. Reads a prompt, writes files, executes them.
holds
- A placeholder string that authenticates nothing
can reach
- broker (by DNS name, over the locked network)
- tunnel (CONNECT only, and only with a token)
cannot reach
- The internet, over any protocol
- The host machine and anything bound on it
- The package registry, source forges, cloud metadata
- provider.example directly
if this process is fully compromised
An attacker who owns this process completely gets a placeholder credential, a proxy that only forwards two enumerated paths, and a tunnel that only dials approved hosts. There is nothing here to steal and nowhere to send it.
The part worth staring at is the wall. Two processes live inside it, and those two are the complete list of things touching both networks — which makes them the complete list of things that could carry anything out. Keep it short and keep every entry boring: a proxy that speaks one API, a relay whose forwarding target is a frozen literal. Anything interesting in the wall is a liability, because “interesting” means “has behaviour somebody could reason their way into abusing.”
Containers are incidental, by the way — they are just the cheapest hard boundary most teams already have. Separate VMs, separate machines or a managed sandbox make the same argument, sometimes better. What must not change is that something other than intent enforces it.
Two settings, two different holes
“The runner has no route out” is easy to say and easy to get wrong, because there are two distinct paths off a container network and the obvious setting closes only one. The first is routing through the host to somewhere else. The second is reaching the host itself — not forwarding at all, but a packet whose destination is the host’s own address on the bridge. Step through the three states and watch the probes.
Marking the network internal removes its default route, so nothing can be forwarded through the host to the internet. Most people stop here. But reaching the host itself is a different path — the packet's destination is the host's own address on the bridge, not somewhere beyond it — and this setting never touched it.
probes, from inside the runner
$ curl https://provider.example/v1/messagescurl: (6) Could not resolve host$ curl http://172.17.0.1:11434/api/tagsHTTP/1.1 200 OK ← still openThe middle position is the one to sit with, because it feels safe and is not. What makes the third position hold is that the result is not a rule: no filter to evaluate, no ordering to get wrong, no earlier entry shadowing a later one. The kernel has no route, so the packet fails at the point of asking.
Container runtimes accept unknown network options, store them, and ignore them. An older daemon takes the gateway-isolation setting without complaint, boots cleanly, and leaves the host path open while your config still claims it is closed. Gate the runtime version and refuse to start below it.
A network with no gateway cannot publish a port — that needs the gateway you just removed. So inbound gets its own process: a minimal TCP relay on both networks whose forwarding target is frozen at deploy time, so reaching it just loops back to the runner.
Notice how much of this is consequence rather than design. Close egress and inbound breaks, so you add a relay — and now something else straddles both networks and has to be argued about. Each control creates the next problem, which is why that list is the thing to audit.
Six things the runner might try
Two of these are the job. Four are what an incident looks like. Send each one, then lift the lockdown and send it again — the second run matters more, because that configuration is not a strawman. It is the default.
what the runner is attempting
Run a model turn. This is the job.
- runnerThe agent process asks its HTTP client for the destination.
- routeThe kernel looks for a route to that address. No gateway on this network means no route, and the call fails here — before DNS, before a socket opens.
- brokerThe credential proxy accepts the connection. It is the only neighbour that can reach anything outside.
- allowlistThe requested path is matched against an enumerated list, and the upstream origin is pinned. Anything not named is refused flatly.
- destinationThe request leaves, carrying the real credential the broker substituted in. The runner never saw it.
pick a destination, send it, then lift the lockdown and send the same one again
The controls are layered and independent, and none is sufficient alone. The missing route stops most attempts before a packet exists, but says nothing about a request to the one host you deliberately allowed. The allowlists are precise but only apply to traffic that already reached something willing to evaluate them. Together they close each other’s gaps; picked individually, each has a wide-open flank.
Injecting a credential without becoming an open relay
The split itself is unremarkable: the runner gets a placeholder, the broker gets the real value from a mounted file, and the runner is pointed at the broker.
services:
runner:
environment:
# Satisfies the client library's "a key is present" guard and nothing
# else. It authenticates against no service anywhere.
API_KEY: placeholder-not-a-credential
# Every independent reader of the provider API needs its own redirect.
# There is almost never just one.
PROVIDER_BASE_URL: http://broker:8787/api/v1
SEARCH_API_BASE_URL: http://broker:8787/api/v1
PACKAGE_API_URL: http://broker:8787/api/v1/packages
networks: [locked]
broker:
environment:
# The real credential, reaching exactly one container. Read from a
# mounted file rather than declared inline, so it stays out of
# `docker inspect` and out of the rendered compose config.
API_KEY_FILE: /run/secrets/api_key
secrets: [api_key]
networks: [locked, egress]Find every reader
This is the step that bites. A codebase rarely has one place that talks to the provider — the model client, a search tool, a package fetcher, telemetry and an update checker may each build their own client with their own base URL. Miss one and it either 401s confusingly or reaches out on a path you never audited. Be empirical: run a full session and account for every request in the broker’s log. No log line means it did not go through you.
A proxy that attaches a real credential to whatever it is handed is worse than no proxy — it is an authenticated gateway anything reaching the port can drive. Four checks stop that, and the order is load-bearing.
// Enumerated, always. Never a wildcard and never a bare "/api/v1":
// the account-management endpoints answer 200 upstream with a real
// credential, and they live on the same host as the work you want.
const ALLOWED_SUFFIXES = ["/chat/completions", "/responses"] as const;
const ALLOWED_PREFIXES = ["/api/v1/packages"] as const;
// A literal ".." collapses during URL normalisation before anyone sees it.
// A percent-encoded separator does not: "/api/v1/packages/..%2fkeys" keeps
// its shape, stays under the allowed prefix, and would be forwarded with
// the real credential attached for upstream to resolve as it pleases.
const ENCODED_SEPARATOR = /%2f|%5c/iu;
const forward = (request: Request, upstreamBase: string, realKey: string) => {
const target = new URL(request.url, upstreamBase);
// ORIGIN FIRST, then path. `new URL("//attacker.example/responses", base)`
// is an origin swap that leaves the pathname looking entirely legitimate,
// so a path-only check hands the real credential to a host of the
// caller's choosing.
if (target.origin !== new URL(upstreamBase).origin) return forbidden();
if (ENCODED_SEPARATOR.test(target.pathname)) return forbidden();
if (!isAllowedPath(target.pathname)) return forbidden();
const headers = setAll(
// `host` misroutes behind a CDN if forwarded verbatim; the length is
// stale once the body is re-encoded. Let the client re-derive both.
removeMany(request.headers, ["host", "content-length"]),
{
// Ask upstream not to compress. The response leg copies headers
// through, so a surviving `content-encoding: gzip` would ride back
// over bytes that were already decompressed, and any
// compression-aware caller dies on it.
"accept-encoding": "identity",
// The only mutation this proxy makes.
authorization: `Bearer ${realKey}`,
},
);
return fetch(target, { body: request.body, headers, method: request.method });
};Allowing the whole API surface on the correct host is tempting and wrong. Account and billing endpoints live on that same host and answer happily to the same credential, so a wildcard turns the broker into a fully authenticated console. Enumerate the handful of paths the runner needs; refuse everything else flatly.
Injection has a return leg, and some APIs echo what they were sent — in an error body, a debug field, a reflected header. If the credential comes back through the broker unfiltered it lands in the runner, and everything upstream of that was wasted. Scrub the response for the values you injected, and fail loudly on a match rather than quietly redacting.
What this costs you
The costs land squarely on the workflows that made an agent appealing in the first place. Better to know now than during an incident review.
The runner cannot install a dependency for code it just wrote, or clone anything. On a vehicle built for authoring code, that is the big one. Bake everything the runtime needs into the image — and test those pinned versions against what the runtime actually looks for, because a stale pin silently falls back to a fetch at run time, which only fails once egress is closed.
A tool whose job is fetching a model-chosen URL cannot coexist with a single way out; covering it would make the broker an open relay. Keep it registered rather than deleting it, answering with a plain explanation — a tool that vanishes or throws a network error reads as a transient fault, and the model will retry into the wall until the turn is spent.
Telemetry, update checks and catalogue pickers tend to hardcode an endpoint with no override. Under a lockdown these can only fail, so switch them off explicitly rather than leaving background requests retrying. Audit the whole tree, not just what fires today: a path that reaches one later fails closed, correctly, but in a way nobody has seen before.
And what it does not do
Not a defence against the host operator
Anyone who can inspect the containers or shell into the broker has the credential. This sits between the agent and the secret, not between an administrator and it.
Does not survive a compromised broker
The broker is the concentration of trust by design. Its defence is being small enough to read in one sitting and running nothing the untrusted side influences. Add a plugin system and the boundary is gone.
Does not stop misuse of legitimate access
The runner cannot steal the key, but it can make as many perfectly allowed requests as it likes — which is your money. Containment is not a budget; rate limits are a separate control this makes easier, not unnecessary.
A passing denial test proves nothing
A test asserting something cannot happen passes when the thing is genuinely blocked — and also when it was never possible in the first place. A green suite of denials can be describing a system with no controls at all. The only way to tell is to remove the control and confirm the test goes red. Run it once locked, once stripped, and sort every assertion into two piles.
| Assertion | Fails when unlocked? | What it is worth |
|---|---|---|
| provider.example reached directly | evidence | Put the runner back on a bridged network and this succeeds immediately. Real evidence. |
| An arbitrary third-party address | evidence | Same. The clearest single demonstration that no route exists. |
| UDP to an external resolver | evidence | Covers the assumption that only TCP was ever considered. Goes red when unlocked. |
| A service bound on the host | evidence | Goes red specifically when gateway isolation is removed — the one assertion that catches that setting being silently ignored. |
| The cloud metadata endpoint | guard only | On a developer machine there is no metadata service, so this passes locked or unlocked. Keep it as a fail-closed guard for cloud hosts; do not count it. |
| IPv6 egress | guard only | With IPv6 disabled on the bridge, that path does not exist either way. A guard against a future change, not proof of a current control. |
Prove it from the untrusted side
Verifying injection by checking the broker’s logs tests your logging. Do it the other way around: stand up a destination that echoes back the Authorization header it received, and put it on the outward network only, so arriving there at all proves the request went through the broker rather than around it. Then assert from inside the runner — it sends a placeholder and reads back the real value. The test never has to trust the broker’s own account of what it did.
Two more things worth automating
Gate the container runtime version and refuse to start below it, since the setting that closes the host path is accepted and ignored by older daemons — a stack that boots cleanly while the boundary is absent is the worst outcome available. And account for every request in the broker’s log during a full session: if something completed and there is no log line for it, it did not go through you, and you have just found the reader you missed.
Design for the agent you cannot vouch for
Most conversations about agent safety happen at the model layer — better prompts, better refusals, better tool descriptions. Worth having, but they all ask the component whose behaviour is uncertain to enforce its own limits. These controls are indifferent to what the model decides: a missing route does not care how persuasive the input was.
Know which layer holds the secret
TCP moves bytes, HTTP gives them meaning, TLS seals HTTP inside TCP. The credential lives in HTTP, which is why whoever terminates the TLS can read it — and replace it.
Trust is a decision, not a property
A certificate chain proves validity, not honesty. Interception is authorised by the trust store, so the real boundary is who can read the root's private key.
Pick the proxy that matches your adversary
A tunnel protects the payload from the middle. A terminating proxy protects the credential from the client. When the client is the untrusted party, only the second one is useful.
Make the separation structural
Different process, different container, different network namespace. The kernel enforces what a code review comment cannot, and each component gets exactly one boring job.
Leave no route, and prove it
Closing egress takes two settings, not one, and the result should be an absent route rather than a rule. Then verify by removing each control and watching the test go red.
Separate the process that holds the credential from the process that runs untrusted code, and give the untrusted one nowhere to go.
Everything else in this article is bookkeeping in service of that one sentence.
None of these primitives are new. TLS termination, private certificate authorities, credential-injecting proxies and isolated network segments have been standard practice anywhere with an auditor for decades. What is new is the reason: we have started running processes whose instructions are partly written by whatever they happened to read. The right response is not a novel one — it is to notice we already know how to contain a component we cannot vouch for, and go and do it.
Questions?
Got a question about something I wrote? Send me a note.