What happened to LiteLLM on 21 October 2026?
An attacker holding three LiteLLM developers' GitHub CLI tokens ran the project's own release pipeline against a lookalike fork, Berri-AI/litellm. The real repository is BerriAI/litellm, one hyphen away. The output, version 1.107.0, was signed and published through the normal channels. Per the vendor's incident post, seven wheels were live on PyPI from 11:39:51 to about 13:23 UTC. Docker Hub and GHCR tags v1.107.0, latest, main-latest and main-stable pointed at the bad image from 11:46:56. GHCR was cut by 13:51, Docker Hub by 14:20.
The same post lists what stayed clean: the official v1.107.0 source tag on GitHub, the enterprise private registry, and every earlier version.
Scale matters here. Datadog puts litellm at roughly 23 million weekly downloads. StepSecurity, and Wiz as quoted by The Next Web, count about 49,000 downloads of 1.107.0 before removal. This is the project's second supply-chain compromise of 2026, after 1.82.7 and 1.82.8 in March.
Anything that pulled latest in that window got the payload without a human choosing to upgrade.
The gateway holds every key you have
That is why it was worth an 11-day operation. A LiteLLM proxy typically stores provider API keys in its database, a master key, the virtual keys your apps use, a DATABASE_URL, and it runs under a cloud role. Datadog and StepSecurity both found that the stealer queried the proxy's tables of upstream provider credentials when a database was configured. On top of that it took cloud credentials (including instance metadata), SSH and Git/npm/PyPI tokens, and the environment of every readable process. StepSecurity also lists configs of 20+ AI coding tools.
The Next Web, citing Wiz, reports stolen AWS credentials being tested about 4.5 hours later, including listing Bedrock models.
Budgets on virtual keys (we covered them in our cost-control article) cap what an app spends through the gateway. They do nothing for a provider key that has already left the building. Spend limits at the provider, per-environment keys and a narrow cloud role are what bound that loss.
How did the attacker get in, and what changed in the story?
The vendor's account changed within two days, and the second version is the useful one. The 21 October post suspected GitHub CLI OAuth tokens were read from logs of an internet-exposed internal LiteLLM deployment, noting that Claude Code can, in specific cases, embed a user's GitHub CLI token in a response body. The 23 October attack-chain post replaced that. An admin key for the internal LiteLLM Agent Platform was committed to a public repo on 27 May 2026, deleted about three hours later, and never revoked. On 10 October one API call with that key returned every agent's configuration, including an org owner's unscoped GitHub CLI token.
From there it was CI. Throwaway branches ran workflows that posted repository secrets to webhook.site, which yielded two more maintainers' tokens. Release approval had been relaxed days earlier for bugfix backports, so a stolen maintainer token could pass the production gate. 136 days passed between the leak and its use.
The vendor now says some early updates were "later proven to be inaccurate or incomplete." Its 30 October post adds a line we agree with: "no pipeline controls can defend against an attacker who is an organization owner, because the owner owns the pipeline." The controls you own are on the consuming side.
Were you hit? Check from the shell
Do not start with pip. The vendor warns that any Python start in an affected environment re-runs cook.pth and sitecustomize.py, pip included, and Datadog confirms that deleting the litellm package directory leaves those files in place. Work from the shell, on every venv, image and CI runner that could have resolved litellm on 21 October. A /tmp/.pc_once_* marker means the stealer finished (Datadog). The vendor adds a warning: finding nothing does not prove the host is clean.
The table is ordered by how cheap each check is.
Exposure checks for litellm 1.107.0, cheapest first
| Check | Command or source | Where |
|---|---|---|
| Lockfiles resolved 1.107.0 | grep -rn '1.107.0' requirements*.txt uv.lock poetry.lock | Repos, CI caches, internal mirror |
| Startup hooks present | find / -path '*site-packages*' \( -name cook.pth -o -name cook.py -o -name sitecustomize.py \) 2>/dev/null | Every venv, container, runner |
| Loader daemon alive | ls -l /tmp/remote_code_loader.lock plus a detached python process | Hosts, running containers |
| Image digest | docker images --digests | grep litellm against digests in the vendor post | Registries, k8s nodes, mirrors |
| Exfil traffic | DNS and proxy logs for litellm[.]dev | Resolver, egress firewall |
| Cloud misuse | CloudTrail GetCallerIdentity from the instance role near install time | AWS accounts |
What do you rotate, and in what order?
If 1.107.0 ran with outbound internet, the vendor says to assume exfiltration succeeded. Rotate from a clean machine, then rebuild from scratch, which the vendor prefers over reinstalling in place. The order below follows the attacker's observed behaviour, cloud first:
- The instance or task IAM role. Stolen AWS credentials were in use the same afternoon (StepSecurity logs activity at 16:20–17:01 UTC).
- Every provider key stored in the proxy database, plus the master key and all virtual keys.
- The
DATABASE_URLpassword. - Everything else that sat in a process environment on the host: CI tokens, Git, SSH, npm and PyPI tokens, Vault and Kubernetes credentials.
- Rebuild from a pinned digest, and purge mirrors and caches that still hold the wheel.
Step 2 is the expensive one. New virtual keys break every client application at once, so someone has to walk each consumer through the change, and the provider keys behind them may be shared with systems nobody remembers. We plan it as a short, announced outage with an owner per consumer.
One more scan applies even if you never installed 1.107.0. The Cloud Security Alliance recommends that any gateway which stored coding-agent traffic (for example with store_prompts_in_spend_logs switched on) be searched for gho_, ghp_ and github_pat_ prefixes, including downstream tracing tools and backups.
Controls that would have kept 1.107.0 out
Most of these are cheap individually; together they cost upgrade speed. The bad wheels lasted about 1 h 43 min on PyPI, and the March versions roughly 40 minutes (The Next Web). A 24-hour cool-down skips both. We use seven days for direct dependencies and an explicit exemption when a security fix cannot wait.
# pyproject.toml: ignore anything uploaded in the last 7 days
[tool.uv]
exclude-newer = "7 days"
exclude-newer-package = { cryptography = false } # exemption, reviewed
# lock exact versions + sha256 per artifact, fail CI if stale
uv lock
uv sync --locked
# pip shops: pip install --require-hashes -r requirements.txt
# pip 26.2 also has: --uploaded-prior-to P7D
# Dockerfile: digest, never a tag
FROM ghcr.io/berriai/litellm@sha256:<digest you reviewed>
# promote a digest only after it verifies against the NEW key
cosign verify --key cosign.pub ghcr.io/berriai/litellm@sha256:<digest>The cosign line needs care. The vendor says the compromised 1.107.0 images "also carry a valid signature from the previous key", which it now treats as exposed. A signature proves who held the key. It says nothing about intent. Trust old-key signatures only for images published before 1.107.0, check digests, and use the key rotated on 23 October, whose fingerprint the vendor published on 30 October.
The rest, with what each costs:
- Default-deny egress on the gateway. It needs your provider endpoints and little else. A rule like that would have blocked the POST to litellm[.]dev and the hourly dead-drop poll. Cost: every new provider is a firewall change.
- No public admin surface. CSA's wording: gateway admin interfaces "should never be reachable from the public internet." The vendor now puts all internal deployments behind an identity-aware proxy.
- One narrow cloud role per gateway. If it calls Bedrock, it gets Bedrock invoke rights and nothing else.
- Content logging off by default, with secret redaction before persistence when you need it.
Tool servers are a separate problem with their own controls; see MCP servers in production.
What does a PyPI attestation prove?
Less than most teams assume. PyPI's docs describe publish attestations as proof that a file "was published via a specific Trusted Publisher". The source location is a different attestation type (SLSA provenance). The DEV Community analysis of 1.107.0 is blunt: a publish attestation "names the workflow that uploaded the file, not the source it built." Here the legitimate workflow did the upload, so a check pinned to the real publisher "would most likely have passed 1.107.0" (the authors mark that as inference).
The same analysis found that litellm attested only 15 releases (1.83.0 to 1.83.14) and none of the 214 since, that 39.0% of the top 1,000 PyPI projects attest their latest release, and that as of October 2026 "neither pip nor uv verifies attestations at install time." BuildKit provenance on the Docker images did name the fork. The attacker's build writes that field, though.
The vendor's own fix has the right shape: a verifier outside GitHub that compares published wheels, sdists and images with the matching tag and yanks on mismatch. A consumer can do a cheap version of that. Diff the wheel's file list against the previous release; StepSecurity cites extra files in the wheel as one of the confirming signals.
What we would do next time
We would treat the gateway host as a vault: nothing else runs on it, it pulls packages only from an internal mirror with a seven-day delay, images by digest, egress default-deny, and its logs scanned like a secret store. The 136 days are the lesson for the credentials side. Deleting a commit does not revoke a key, so every exposed key gets a revocation ticket that closes only when the provider confirms it. For teams running LiteLLM or another LLM proxy in production, we run a one-week AI gateway and agent supply-chain hardening review that covers pins, digests, egress, key scoping and audit trail.