Security & Infrastructure

Baseten Had Admin Access Sitting in a Public Docker Image for 40 Months. The Vulnerability Class Is the Story, Not the Company.

Baseten Had Admin Access Sitting in a Public Docker Image for 40 Months. The Vulnerability Class Is the Story, Not the Company.

A 2023 Docker Build Credential Sat in a Public Image Until 2026. The Interesting Part Is Not That It Happened.

In July 2026, a security firm called Strix pointed its autonomous pentesting agent at *.baseten.co and walked away 25 minutes later with a live GitHub personal access token carrying admin and push rights to Baseten’s main product repository, their GitOps cluster-configuration repo, and their Homebrew tap. The token had been sitting in a publicly downloadable Docker image since March 2023. Nobody caught it for three years and four months.

Baseten is not a hobbyist side project. The company is valued at $13 billion and hosts inference infrastructure for serious production workloads. When I read the Strix disclosure, my first instinct was the same one most engineers have: how does this happen at a company with real security resources? But that framing misses the more important question, which is structural. The token did not leak because Baseten is careless. It leaked because the Docker build credential pattern that produced this exposure is extremely common, Docker’s documentation on the danger is easy to miss, and Harbor’s default public project behavior is exactly backward from what most engineers assume.

This piece is not a pile-on. It is a technical walkthrough of the exact chain of decisions that produced this outcome, an analysis of why none of the individual steps felt dangerous at the time, and a checklist of what you need to verify in your own infrastructure right now.

The Attack Chain, Step by Step

The Strix writeup is worth reading in full, but I want to reconstruct the chain because each link matters independently.

Step 1: Surface Enumeration Found a Forgotten Subdomain

Strix started with certificate transparency logs and host enumeration. It found a Harbor container registry at gcp-us-east4-zlw.registry.baseten.co. This subdomain is almost certainly not in anyone’s mental model of “the attack surface.” The people who knew about it when it was set up in 2023 may no longer work at Baseten. The people who do work there now probably don’t think of it as an externally accessible endpoint.

This is the first structural lesson. Infrastructure accretes. A Harbor instance that was “just for internal builds” in 2023 becomes, three years later, a subdomain that nobody is monitoring, nobody is scanning, and nobody has thought to include in access reviews. Certificate transparency logs see every TLS certificate you issue. Attackers use them. Your own security tooling often does not.

Step 2: Harbor’s Default Project Visibility Was Public

Harbor organizes container images into projects. By default, Harbor allows creating public projects. A public project in Harbor means that anyone — no authentication, no token, no account — can list repositories in that project, get anonymous pull tokens, and download image manifests and blobs. The Baseten Harbor instance had at least one public project containing a repository called baseten/baseten-app.

I have set up Harbor deployments. The first time you create a project, the UI defaults to public. If you are moving fast and the Harbor instance feels internal — it is behind a corporate subdomain, you are not thinking about it as a public endpoint — you do not notice this setting or you notice it and think “fine for now, I’ll change it later.” Later never comes.

Step 3: The Image Contained Dead Credentials, Then Live Ones

Strix pulled the image and ran TruffleHog against it. It found a pair of AWS keys first. Those were dead — InvalidClientTokenId on a test call. At this point, a human pentester might have written “exposed AWS credentials (revoked)” in their report and moved on. Strix kept looking.

It found a GitHub personal access token in history[].created_by — the Docker image config’s build history field, not the filesystem layers. This distinction is critical and I will come back to it. The token was for an account called basetenbot. It was still valid.

Step 4: The Token Had Admin Scope on Production Repositories

A GitHub PAT with zero permissions is a dead end. Strix checked the OAuth scopes on the token: repo. Full repository access. Then it enumerated the specific repositories basetenbot had access to and what level of access. Three repositories had admin: true, push: true. Based on context in the disclosure — the product repo, the GitOps repo driving their clusters, the Homebrew tap — this token could have pushed malicious code to Baseten’s core product, modified the infrastructure configuration that controls their production clusters, and poisoned their Homebrew distribution channel. Four more private repositories had read/write access.

The token was minted in March 2023 and never rotated. It had been valid for 40 months when Strix found it.

The Docker Build History Problem Is Not Obvious

I want to spend time on the specific technical mechanism because it surprises experienced engineers who know perfectly well not to commit secrets to source control.

A Docker image has two separate stores of information. The first is the filesystem — the actual files in each layer that get mounted when you run a container. The second is the image config, which is a JSON document that records metadata about the image and its construction. The config’s history array contains one entry per build step, and each entry has a created_by field that records the instruction that created that step.

When you write a Dockerfile like this:

ARG GITHUB_TOKEN
RUN GITHUB_TOKEN=${GITHUB_TOKEN} bash -c '\
  if [[ "${GITHUB_TOKEN}" != "" ]]; then \
    git config --global --add \
      url."https://${GITHUB_TOKEN}@github.com/".insteadOf "[email protected]:"; \
  fi'

Docker records the RUN instruction in history[].created_by with the ARG value already substituted. The token value is baked into the image config. It is not in the filesystem layers. Running docker history without --no-trunc truncates the output and you will not see it. Scanning image layers with most tooling will not catch it because most tooling scans files, not the config blob.

Docker has documented this behavior for years. The canonical fix is BuildKit secret mounts:

RUN --mount=type=secret,id=github_token \
    GITHUB_TOKEN=$(cat /run/secrets/github_token) \
    git config --global --add \
      url."https://${GITHUB_TOKEN}@github.com/".insteadOf "[email protected]:"

A secret mount does not persist the value into the image config or the filesystem. It exists only in the build context of that specific RUN step. But BuildKit secret mounts were not available until Docker 18.09 and were not well-documented in practical build workflows until considerably later. A Dockerfile written in early 2023 by an engineer who needed to solve “how do I fetch a private dependency in CI” would not have had secret mounts as a readily available mental model. The pattern Baseten used — pass the token as a build ARG — is the pattern you find in Stack Overflow answers and blog posts from that era.

There is also a second problem that the Strix disclosure correctly flags. Even if you get the token value out of the build history, if you use git config --global to write an authenticated URL, that URL is stored in the .gitconfig file inside the image filesystem. Cleaning the credential from the history still leaves a copy in the layers. You need to fix both, and then you need to make sure you have actually removed or expired the old images — changing the Dockerfile does nothing to images that were already pulled and cached.

Why This Took 40 Months to Find

The more interesting question is not “how did this happen” but “why did it persist.” Three years and four months is a long time. Baseten presumably ran security reviews in that window. They presumably had engineers who knew about Docker build credential risks. So why did nobody catch this?

The answer has several components.

The image was not on the obvious attack surface. Engineers auditing Baseten’s security would look at their APIs, their authentication flows, their database access controls, their network configuration. A Harbor instance on a regional GCP subdomain is not in the standard security review checklist. It is infrastructure that exists below the level of abstraction where most security thinking happens.

The project was public by default and nobody revisited it. Harbor’s default-public setting created a situation where the registry was effectively open to the internet, but nobody thought of it that way because it was behind a corporate subdomain. The mental model of “internal infrastructure” did not match the technical reality of “publicly accessible Harbor project.”

The token had not caused any visible problems. If someone had used the token to push malicious code, Baseten would have noticed. But nobody used it — or if they did, they were careful not to create evidence. A credential sitting idle in an image is invisible. There are no access logs being checked, no anomalous API calls to alert on, no observable signal that anything is wrong.

Credential rotation was not automated or enforced. The token was created in March 2023 and never rotated. A well-run secrets management system would have caught this: tokens should have expiry dates, rotation should be automated or at minimum enforced by policy, and any token that has not been rotated in N months should generate an alert. GitHub now supports fine-grained PATs with mandatory expiry. The classic PAT used here had no enforced expiry.

The Comparative Landscape: This Is Not a Baseten Problem

It would be easy to read this story as “Baseten did something wrong.” That is not the right frame. The right frame is that this vulnerability class is endemic across the industry, and the fact that it takes an autonomous agent to find it tells you something about the gap between how organizations think about their attack surface and what their attack surface actually is.

Vulnerability Class Why It Persists Standard Detection Method Why Detection Fails
Build credentials in Docker history BuildKit secret mounts not widely used pre-2023; ARG pattern is the Stack Overflow answer Image scanning (Trivy, Grype) Most scanners check layers, not image config history
Public Harbor projects Default-public setting; infra feels “internal” Manual configuration review Forgotten subdomains not in review scope
Long-lived machine tokens No expiry enforcement; token works, don’t touch it Secret rotation audits Rotation audits cover known secrets, not secrets baked into images
Overprivileged build accounts Build needed one private dependency; token had broader scope “for convenience” IAM review Machine accounts are underweighted in access reviews

I have personally seen the credentials-in-Docker-history pattern at three different companies I have worked with or consulted for. In each case, the engineers who wrote the Dockerfiles were experienced. The problem is not ignorance — it is that the mental model of “what gets baked into an image” is incomplete in a specific and non-obvious way.

The industry has been slow to fix the tooling gap. Trivy and Grype are the standard open-source image scanners. Both are excellent at finding CVEs in installed packages and checking filesystem layers for secrets. Neither, as of mid-2026, scans the image config’s history field by default. TruffleHog, which Strix used, does support scanning Docker image history — but TruffleHog is not the default tool in most CI pipelines.

The AI Security Agent Angle Is Real, But Oversold

Strix is selling an autonomous pentesting agent, and this disclosure is marketing as much as it is public service. I say that not to dismiss it — the disclosure is technically accurate and responsibly handled — but to be clear-eyed about what the “AI did this in 25 minutes” framing does and does not mean.

What it means: an AI agent can do systematic recon that a human would find tedious. Enumerating cert transparency logs, mapping subdomains, checking Harbor project visibility, pulling images, running TruffleHog, testing credentials — each of these steps is mechanical. A determined human pentester would do all of them. The agent compresses the time and removes the boredom that makes humans skip steps.

What it does not mean: this attack was sophisticated. It was not. The individual steps were all well-documented techniques. The vulnerability had been in place for 40 months. A motivated human with a checklist would have found it. The reason it persisted is not that the attack required AI — it is that nobody ran the checklist.

The more important observation is that AI attackers will run these checklists at scale. If an autonomous agent can scan Baseten in 25 minutes and find a critical credential, the same agent or a similar one can scan hundreds of targets in parallel. The threat model for infrastructure security needs to incorporate the idea that systematic, boring, checklist-driven attacks are now cheap and scalable in a way they were not three years ago. That changes the calculus for what counts as “unlikely to be found.”

What Baseten Got Right

The disclosure timeline is worth reading carefully as a model of how to handle this correctly.

Strix reported at 11:10 PM on July 13. By the next morning, Baseten had made the Harbor project private. When Strix flagged that the token itself was still live, Baseten confirmed the issue as critical and rotated the token by 4:34 PM the following afternoon — less than 18 hours from initial report. They closed out lower-severity findings within the week. They cooperated on the public disclosure timeline and reviewed a draft of the post before it went live.

I have seen companies take weeks to rotate a token after a responsible disclosure report. I have seen companies dispute the severity of findings to avoid escalation. Baseten did neither. Their security team treated this as what it was — a critical issue requiring immediate action — and they moved quickly. The fact that a credential sat in an image for 40 months does not tell you much about how a company responds to incidents. The response tells you more.

The Checks You Should Run Right Now

If you run containers and use GitHub, there are four specific things to verify.

Check what your Harbor instance exposes without authentication. Log out entirely, then browse to your instance. What can you see? What can you pull? Many engineers have never done this because they are always logged in when they use the registry. The unauthenticated view is the attacker’s view.

Inspect your image build histories. For any image that has ever been built with a token or credential as a build ARG, run:

docker history --no-trunc your-image:tag

Or inspect the raw config blob directly:

docker inspect --format='{{json .Config}}' your-image:tag | python3 -m json.tool

Look at the full layer history. If you see credential values anywhere, rotate the credentials immediately — even if the image is no longer in active use. Pulled copies may exist in caches anywhere.

Audit your machine account permissions. basetenbot needed to fetch private dependencies. It got repo scope on the entire organization and admin rights on production repositories. For dependency fetching, you need read access to the specific repositories that contain the dependencies. Nothing more. Review every machine account and bot token in your GitHub organization and ask: what does this account actually need? Fine-grained PATs exist specifically to enforce the minimum necessary permissions.

Add token expiry to your machine accounts. Classic GitHub PATs have no enforced expiry. Fine-grained PATs require an expiry date. If you are using classic PATs for machine accounts, migrate them to fine-grained PATs with a rotation schedule. A credential that can persist for 40 months without expiring is a credential that will eventually be found.

The Systemic Problem: Security Reviews Miss Infrastructure Below the API Layer

The Baseten disclosure illustrates a gap that I think is underappreciated in how most companies structure their security work.

Most application security reviews focus on the application: APIs, authentication, authorization, input validation, session management — the things visible to users with well-established testing methodologies. Infrastructure security reviews tend to focus on cloud configuration: IAM policies, security groups, S3 bucket permissions, database network access.

The gap is the layer in between: the build and deployment pipeline infrastructure. Container registries. CI/CD credentials. Artifact storage. Package repositories. These systems are “internal” in the sense that engineers use them, but they are often not internal in the technical sense of being network-restricted. And the credentials that flow through these systems — build tokens, deployment keys, registry credentials — often have high privilege because build and deployment processes need high privilege to do their jobs.

This gap exists at most companies I have worked with. The application security team does not own the build infrastructure. The infrastructure team does not think of the build pipeline as an attack surface. The result is a category of high-value targets that are systematically under-reviewed.

The Baseten Harbor instance is a perfect example. It is not an application and not cloud infrastructure in the IAM sense. It is a container registry running on a GCP subdomain with a public project. Nobody put it on the checklist because the checklist was designed for a different mental model of what the attack surface looks like. That is a process problem, not a Baseten problem.

The Broader Credential-in-Image Ecosystem

The Docker build history vector is one of several ways credentials end up in container images. The full taxonomy is worth knowing.

Build ARGs (this incident): Credentials passed as build arguments get recorded in history[].created_by. The fix is BuildKit secret mounts. The detection gap is that most scanners check filesystem layers, not the config blob.

ENV statements: Setting a credential as an environment variable in a Dockerfile bakes it into both the image config and every subsequent layer. This one is more widely known and more likely to be caught by scanners.

Intermediate layers not squashed: Even if the final layer of an image does not contain a credential, an intermediate layer might. If an engineer runs COPY credentials.json /tmp/ && RUN pip install ... && RUN rm /tmp/credentials.json, the file is present in the intermediate layer and recoverable from the image. Multi-stage builds with proper secrets management are the fix.

git clone with embedded credentials: Cloning a private repo with an authenticated URL during a build can leave the credential in the .git/config file in the layer, or in the shell history of the RUN step.

pip/npm/cargo credentials: Package managers that authenticate with tokens sometimes write those tokens to local configuration files during a build step. If those configuration files end up in the image, so do the tokens.

The common thread: any credential that touches the build process has multiple paths into the image. BuildKit secrets, multi-stage builds with explicit secret-free final stages, and post-build scanning of both layers and history are the defensive controls. All three are necessary. Any single one is insufficient.

The Rotation Problem Is Harder Than It Looks

Let me say more about token rotation because the standard advice — “rotate your secrets regularly” — obscures how hard this is in practice for machine accounts embedded in build systems.

A human user credential is relatively easy to rotate: you change the password, the user logs in again, done. A machine account credential that is embedded in a Dockerfile, referenced in CI configuration, stored in a secrets manager, and consumed by three different build pipelines is a coordination problem. Rotating the token requires updating it in every place it is used, simultaneously, without breaking ongoing builds. If you update the token in the secrets manager before updating the Dockerfile reference, builds fail. If the token is hard-coded in a Dockerfile rather than injected at build time, the rotation is even harder — you have to change the Dockerfile, rebuild all affected images, and redeploy everything that uses those images.

This coordination cost is why machine credentials rot. Engineers know they should rotate them. They also know that rotating them requires careful coordination across multiple systems, creating a window where builds might fail, and potentially waking someone up at 2 AM when a pipeline breaks. So the rotation gets scheduled for “the next maintenance window” and then bumped because something more urgent came up, and then forgotten.

The industry answer to this problem is secrets managers with automatic rotation: HashiCorp Vault, AWS Secrets Manager, Google Secret Manager. These tools can rotate credentials automatically and provide a stable reference (a Vault path, an ARN) that consumers update from automatically. The token value changes underneath the reference without requiring manual coordination across every consumer.

The catch is that using secrets managers correctly requires discipline in how credentials are consumed. If your Dockerfile bakes the token value into an ARG at build time rather than fetching it at runtime, automatic rotation in the secrets manager does nothing — the old value is still in the image. The secrets manager is only effective if consumers are reading the current value at the time they need it, not at build time.

In the Baseten case, the token was baked into the image at build time in March 2023. Even if Baseten had started using a secrets manager in 2024 and rotated all their credentials, this specific token — the one in the image config of a three-year-old build — would not have been touched by that rotation. The old image was still out there, still had the old token, and the old token was still valid because it had not been explicitly revoked.

This is the deepest lesson from this incident: the rotation of a credential and the revocation of old instances of that credential are separate problems. Rotation creates a new token. Revocation invalidates the old one. You need both. A rotation that creates a new basetenbot token but leaves the old one active accomplishes nothing for an attacker who already has the old one.

GitHub’s fine-grained PATs are better here because they have enforced expiry dates. A token with a 90-day expiry that was created in March 2023 would have expired by June 2023. If Strix had found this image in July 2026, the token would have been dead. Expiry is not the same as rotation, but it provides a ceiling on how long a compromised credential remains valid. The classic PAT has no such ceiling.

What This Means for Supply Chain Security

The Baseten exposure deserves context in the broader conversation about software supply chain security, which has been a major theme in the industry since the SolarWinds compromise in 2020 and the Log4Shell disclosure in 2021.

Supply chain attacks typically work by compromising a component that many downstream systems trust and use. The scenario with the Baseten token is not exactly a supply chain attack — Strix found it before anyone appeared to exploit it — but it illustrates the pathway. An attacker with admin access to Baseten’s product repository could push a change to the ML inference library or runtime that all Baseten customers depend on. That change would look like a legitimate update from a trusted source. Downstream customers would pull and run it.

The Homebrew tap is a specific vector worth noting. Homebrew is the de facto package manager for macOS developers. A Homebrew tap is a repository of formulae — instructions for installing software. If an attacker controlled Baseten’s Homebrew tap, they could modify the installation formula to deliver malicious binaries to any developer who ran brew install or brew upgrade for a Baseten tool. That is a supply chain attack with a large potential blast radius, and it would have been within reach of whoever held the basetenbot token.

Baseten’s customers trust Baseten to secure their own infrastructure because a compromise of Baseten’s infrastructure is a potential compromise of anything those customers have deployed through Baseten. When a company is valued at $13 billion and handling inference workloads for “serious production applications” (as their own marketing says), the security of their build pipeline is not just their problem. It is a problem for everyone who depends on them.

Falsifiable Predictions

First: within the next 18 months, at least two major container image scanning tools — Trivy, Grype, or Snyk Container — will add explicit scanning of Docker image config history as a default behavior, driven by disclosure reports like this one. The tooling gap is documented; the fix is not technically difficult; the pressure to close it will increase as AI-assisted scanning makes this class of finding more common.

Second: GitHub will deprecate classic PATs for organizational machine accounts within two years. Fine-grained PATs with mandatory expiry already exist. The remaining friction is migration cost, and GitHub has been progressively tightening access controls. The security argument for removing unbounded, overprivileged, non-expiring tokens is strong enough that the deprecation is a matter of when, not if.

Third: Harbor will change its default project visibility to private within the next major release cycle. The current default — public — is the wrong default for a system that most teams treat as internal infrastructure. The Baseten case is documented and specific enough to serve as the forcing function for that change.

The credential sitting in your own image right now is not hypothetical. The build pipeline you set up in 2023 with an ARG-based token and a Harbor project you forgot to lock down is not an edge case. Go look before someone else does.

Was this analysis useful?
Michael Sun
Michael Sun

Solo founder and engineer writing opinionated, benchmark-driven analysis of AI, security, and developer tooling.

About ThesisBench →

Discussion

Leave a comment

Comments are moderated and appear after review. Be specific — vague praise and drive-by hot takes are equally likely to be skipped.

Related