Independent ecosystem audit
Elixir's retrieval gap is a rollout problem, not an access problem.
In August 2026 Evil Martians published a scorecard of 93 Ruby ecosystem resources, measuring whether AI agents can actually reach Ruby's documentation. On the announcement thread, someone asked for the same study for Elixir. This is that study — and the answer is not the one the Ruby findings would lead you to expect.
In brief
Five findings
-
Elixir is not blocked. 98% of resources allow AI crawlers and exactly one applies an AI-specific block. The crawler-unblocking campaign Ruby needs would be wasted effort here.
-
The tooling already exists; the rollout does not. ex_doc has emitted markdown and
llms.txtby default since January 2026, yet only 26% of ex_doc packages expose them — 69.3% of download volume is unreachable as markdown. -
Three distinct causes, all mechanical. Stale documentation builds (59% of volume), an explicit one-line opt-out in
mix.exs(6%), and a hardcoded default inrebar3_ex_docthat excludes every Erlang package. -
Elixir built the mechanism Ruby is still asking for.
usage_rulesdelivers version-pinned guidance straight into an agent's context with no crawler involved — and only 3.5% of repositories use it. -
Retrieval advantage does not survive the quality gate. Elixir leads Ruby on four of five retrieval criteria, yet ties it exactly on training-corpus quality. Being reachable and being retained are different things.
Section 01
Three layers, not two
Ruby's scorecard measures whether an agent can fetch your docs, and whether they reach the model's weights. Applied to Elixir, that framing produces a table of green checks and misses the layer Elixir is strongest in.
Retrieval
Can an agent fetch your documentation at request time?
Training
Will your documentation reach the model's weights at all?
Injection
Do your rules arrive in the agent's context with no fetch and no training?
Layer 3 dominates for a library, and the reason is structural rather than aesthetic: it is
pinned to the dependency version that was actually resolved. Layer 1 serves whatever
latest says. Layer 2 serves whatever was true when the crawl ran, routinely a
year ago.
Only Layer 3 can tell an agent “you are on Phoenix 1.8, here are 1.8's rules.”
That is precisely the failure every Elixir developer recognises when a model writes Phoenix 1.7 idioms into a 1.8 project. It is a Layer 3 problem, and no amount of Layer 1 work will fix it.
Section 02
Elixir leads Ruby on four of five retrieval criteria
Eighty-five curated ecosystem resources — core language and runtime docs, frameworks, libraries, tooling, community sites, learning platforms and blogs — scored on the same six criteria Ruby used.
| Criterion | Elixir (85) | Ruby (93) | Verdict |
|---|---|---|---|
| robots.txt allows AI | 83 · 98% | 87 · 94% | Elixir ahead |
| Crawlable (AI agent actually served) | 81 · 95% | — | not scored by Ruby |
| Sitemap | 62 · 73% | 56 · 60% | Elixir ahead |
| llms.txt | 33 · 39% | 51 · 55% | Elixir behind |
| Content negotiation | 26 · 31% | 16 · 17% | Elixir ahead |
| .md routes | 27 · 32% | 23 · 25% | Elixir ahead |
The shape of the single gap is more informative than the gap itself.
Ruby's wins are hand-written. Elixir's are generated. Each ecosystem did the work the other didn't.
llms.txt is a file a human sits down and authors, and 55% of Ruby projects did
exactly that. Content negotiation and .md routes arrive free from ex_doc, with
nobody deciding anything at all.
Elixir does not need Ruby's crawler-access campaign. Exactly one resource
blocks AI agents specifically: r/elixir, where a browser receives 200 and GPTBot
receives 403. Two disallow AI agents in robots.txt: Exercism and r/elixir. Ruby
had six blockers, including RubyDoc.info refusing ccbot, gptbot, claudebot and
google-extended.
A separate audit, run over its own 93-resource list, measured the training-corpus layer and found Elixir clearing the FineWeb-Edu quality gate on 19/93 resources — exactly tied with Ruby. Leading on four of five retrieval criteria does not mean the corpus keeps your pages. Section 05 covers this.
Mean score by tier
Score out of 6 · the further from the core, the worse it gets
The community tier scoring worst is the expensive one. Forums, Elixir School, Exercism and the podcasts hold the idiomatic, problem-shaped answers — the content that teaches a model to write a language well rather than merely to name it. It is the least retrievable tier in the ecosystem.
Everything Elixir core publishes through a current ex_doc scores a perfect 6/6: the language docs, getting-started, Mix, IEx, Logger, EEx, ExUnit, ExDoc. The toolchain works. That is the whole point.
Section 03
Three causes, all of them mechanical
ex_doc v0.40.0, released 20 January 2026, introduced the Markdown formatter and generates
llms.txt by default. This is precisely the fix Evil Martians ask of rdoc —
“one change that lifts every gem at once.” Elixir shipped it seven months ago.
| Classification | Packages | Downloads / 90d | Volume | Fix |
|---|---|---|---|---|
| Compliant — llms.txt + markdown | 68 | 103,819,142 | 28.8% | — |
| Cause A — explicit formatter opt-out | 19 | 21,175,906 | 5.9% | one line |
| Cause B — stale docs (ex_doc < 0.40.0) | 170 | 212,913,511 | 59.2% | republish |
| Not ex_doc | 26 | 21,998,591 | 6.1% | n/a |
| Total documented | 283 | 359,907,150 | 100% |
Cause A — an explicit list silently opts you out
Any project that pinned a formatters: list before January 2026 keeps exactly that
list. The new default never applies. Nothing warns you, because the option was always legal
and still is. The docs build, they look correct, and they are invisible.
defp docs do
[
formatters: ["html", "epub"],
formatters: ["html", "markdown", "epub"],
...
]
end
Fourteen of the nineteen were confirmed by reading mix.exs directly, including
phoenix, oban, credo, guardian,
oauth2, reactor and stripity_stripe. The controls —
ecto, plug, req, elixir — set no formatters: key at all, inherit the default, and
score. Same ex_doc version, opposite outcome, one line of difference.
Phoenix is the highest-volume opted-out package, at 4,024,850 downloads per 90 days. The pin predates the Markdown formatter, so this reads as inertia rather than intent — which is exactly why a one-line change fixes it.
Cause B — stale docs, no code change required
170 packages simply have not republished documentation since January 2026. This is
dependency-graph bedrock: telemetry (5.67M downloads/90d), mime
(4.78M), decimal (4.76M), hpax, nimble_options,
phoenix_pubsub, websock. Packages an agent meets constantly and can
never read as markdown.
“Isn't this just stale docs?” — mostly yes, and that is checkable. llms.txt is baked into the docs tarball at build time, not served by the
platform: 0 of 225 packages built with a pre-0.40 ex_doc serve it, while
75 of 94 built with a current ex_doc do. A republish genuinely fixes
225 of the 244 non-compliant packages — 92%. The remaining 19 are not
explained by age. That is the whole argument for ordering the republish sweep above the pull
requests.
Cause C — the Erlang side is excluded upstream
Splitting the same 283 packages by build tool exposes a cause invisible in the aggregate.
| Build tool | Packages | Compliant | Rate | Downloads / 90d |
|---|---|---|---|---|
| mix | 201 | 65 | 32% | 268,912,475 |
| rebar3 | 66 | 2 | 3% | 85,477,837 |
rebar3_ex_doc hardcodes its default formatter list to
["-f", "html", "-f", "epub"], and its --formatter help text offers
only “html” or “epub” — markdown is never mentioned. Every Erlang package documented
through the plugin is excluded regardless of which ex_doc version it pulls in.
Two packages escape by passing the formatter explicitly. observer_cli sets
{ex_doc, "--formatter html --formatter markdown --formatter epub"}.
That workaround is proof the capability is present and only the default is wrong.
One default change in rebar3_ex_doc would lift roughly 64 packages and
about 85 million downloads per 90 days — the Erlang ecosystem's exact equivalent of
the rdoc change Ruby is asking for.
Section 04
Elixir built the mechanism Ruby is still asking for
usage_rules has
been shipping since May 2025 and draws 223,557 downloads per 90 days. A package ships
usage-rules.md, or a usage-rules/ directory, and
mix usage_rules.sync consolidates the rules of your actually-resolved
dependencies into AGENTS.md, CLAUDE.md, or agent skills.
Ruby's study lists the equivalent — “establish shared gem/agent convention” and “create a unified Rails MCP server” — under things to build. Elixir has had one in production for fifteen months.
| Signal | Repos | Share |
|---|---|---|
usage-rules.md or usage-rules/ | 10 | 3.5% |
AGENTS.md | 11 | 3.8% |
CLAUDE.md | 9 | 3.1% |
llms.txt committed in the repo | 0 | 0% |
Shipping agent rules today: ash, igniter, reactor,
spark, mimic, nebulex, ex_phone_number,
mdex, sobelow, and phoenix.
Phoenix ships usage-rules/ containing ecto.md,
elixir.md, html.md and liveview.md — and ships no
llms.txt and no markdown output. It is on the wrong side of Layer 1 and the right
side of Layer 3. A pure Layer 1 scorecard grades Phoenix as failing, which is exactly the
misleading result a straight port of Ruby's methodology produces.
Section 05
Reachable is not the same as retained
The training-corpus layer is the most expensive and least actionable part of Ruby's scorecard, and the primary instrument here does not cover it. A separate audit measured it across its own 93-resource list; those results are reproduced with attribution.
| Signal | Elixir | Ruby | Reading |
|---|---|---|---|
Common Crawl CC-MAIN-2026-30 | 89/93 | — | broad presence |
| The Stack v3 (mapped repos) | 77/83 | — | code channel strong |
| FineWeb-Edu quality gate | 19/93 | 19/93 | tied — no advantage |
Elixir's documentation is broadly present in the open crawl. Only about a fifth of sampled pages clear an open training-quality proxy — exactly matching Ruby.
Whatever advantage the retrieval layer confers, it does not survive into corpus retention on this measure. The practical implication is a fourth kind of work, distinct from republishing or one-line changes: improve the first 300–500 words of important guides — lead with a concrete problem and a worked explanation rather than navigation and API chrome — then remeasure with the same classifier.
These three rows come from the separate audit described above and were not re-verified by the instrument behind the rest of this report. Its 93-resource list differs in composition from the 85-resource list used everywhere else, so the denominators must not be combined. Common Crawl, The Stack and FineWeb-Edu are open proxies for what a private model vendor ingests — presence is observational and proves nothing about training.
Section 06
What to do, ordered by leverage
Ordered by reach relative to effort, not by scorecard column. The first three are mechanical and uncontroversial; together they address the large majority of the measured gap.
-
Change the
rebar3_ex_docdefault formatter listAdd
markdownto the default alongside html and epub, and mention it in the--formatterhelp. The highest ratio of reach to effort available, and it fixes the tier currently sitting at 3%. -
Run a documentation republish sweep
Every one of these needs only
mix hex.publish docsbuilt with a current ex_doc. Best done as a coordinated community push, or by having Hex warn at publish time when docs come from a pre-0.40 ex_doc. -
Open the one-line
formatters:pull requestsStart with phoenix, then oban, credo, guardian, oauth2, reactor, rewrite, stripity_stripe, amqp, digital_token, glob_ex, oban_met, oban_web, opentelemetry_absinthe.
-
Publish a site-level
hexdocs.pm/llms.txtAn index of every package's
llms.txt. It does not exist today, and it is the single highest-leverage missing artifact found anywhere in this study — one file that makes the whole package graph discoverable in a single fetch. -
Drive Layer 3 adoption past 3.5%
The strategic one. If
usage-rules.mdwere as ordinary as a README, an agent working in any Elixir project would receive correct, version-pinned idioms with no crawler and no corpus lag. Nothing in Ruby competes with this, and the tooling already ships. -
Raise the opening 300–500 words of the guides that matter
The one action aimed at Layer 2 rather than Layer 1. Elixir ties Ruby at 19/93 on the FineWeb-Edu gate, so reachability is not the binding constraint there. Lead with a concrete problem and a worked explanation instead of navigation and API chrome, then remeasure.
-
Fix the community tier
Elixir School, Thinking Elixir, ElixirWeekly and the forums average 2.08/6 while holding the ecosystem's most idiomatic content. The cheap wins are sitemaps and
.mdroutes, not a rewrite. -
Leave robots.txt alone
98% of resources already allow AI crawlers. Importing Ruby's unblocking campaign would address a problem this ecosystem does not have; the effort belongs in steps 01 to 03.
Section 07
Method, traps and limits
Every figure is a live HTTP probe plus the public Hex and GitHub APIs, collected on 18 August 2026. No API keys, no paid tooling, nothing that cannot be re-run by anyone reading this. Ruby's comparison figures come from the Evil Martians scorecard, measured June 2026.
This is an independent analysis. It is not affiliated with, commissioned by, or endorsed by the Elixir core team, the Erlang Ecosystem Foundation, or any of the projects named in it.
This report is published on phxagents.dev, which ships an Elixir/Phoenix agent-rules package. The disclosed interest is in a conclusion, not a measurement: Section 06 step 05 argues for wider Layer 3 adoption, and that is the category the package sits in. No measurement here touches it. Being a GitHub-distributed Claude Code plugin rather than a Hex package or a documentation resource, it falls outside all three sampling frames by construction, and appears in none of the 85 resources, 283 packages or 287 repositories.
Four measurement traps — each makes the ecosystem look worse than it is
If you reproduce this study, these four will change your answer. Each was hit and corrected during this measurement; before-and-after figures are given so you can tell whether your own run is affected.
- Truncated gzip produces phantom transport errors. Reading a capped number of bytes from a gzip stream leaves it truncated, and a strict decompressor raises an error that is easily caught as a failed request. Uncorrected this reports crawlable 8/85 and sitemap 13/85; corrected, 81/85 and 62/85. Use an incremental decompressor that tolerates partial streams.
- Probing
llms.txtonly at the site root. On documentation hosts the file is namespaced per project (hexdocs.pm/ecto/llms.txt), so root-only probing scores those resources as missing: 7% versus 39% depending on whether every ancestor path is checked. The same applies to content negotiation — a package root may return itsllms.txtastext/plain, so the probe must target a real documentation page or it reports near-zero. - Conflating “blocked” with “dead URL”. Without a browser user-agent control, a 404 or blanket bot-protection reads as an AI-specific block. Three of five apparent blocks here turned out not to be AI policy at all. “Blocked” should mean: a browser is served and the AI agent is not.
- Counting matching lines instead of matches. Sitemaps are frequently emitted as a single line with no newlines, so a naive line count reports one URL where there are 88.
Two tempting explanations are also wrong, and worth ruling out early: the gap does
not track ex_doc version (Phoenix runs a current 0.40.3), and it does not
track the per-package hexdocs subdomain (every package redirects there). The actual cause
only becomes visible on reading mix.exs.
The general rule: a probe that fails closed produces alarming, publishable, wrong numbers — and every trap above fails in the alarming direction. Verify each headline against a hand-measured control before believing it.
Limits of this study
- The 85 resources are a curated list, not a census. Category membership is a judgement call, and adding more blogs would lower the mean.
- The package tier is the top 300 by 90-day downloads. Ranking by all-time downloads gives 22% rather than 26%, because it over-weights dormant Erlang transitive dependencies.
- Layer 3 detection reads the repository root via the GitHub API, so monorepo packages whose rules file sits in a subdirectory are missed. 3.5% is a floor, not a ceiling.
- The Layer 2 figures come from a separate audit over a different 93-resource list and were not re-verified by the instrument used for everything else here. Its denominators are not interchangeable, and its own retrieval columns are superseded by Table 1.
llms.txtand content negotiation are the same signal in Elixir — one formatter emits both, measured at 50/50 agreement on a balanced sample. In Ruby they are independent, because each project solved them separately.- MCP availability (Tidewave, ElixirLS,
usage_rules.search_docs) is named but not measured. - No claim is made here about whether models actually write better Elixir
as a result. That is the outcome metric and the honest next study. Ruby's work does not
measure it either — the widely-quoted “0 out of 1,267 solutions across 13 models” is
Chad Fowler's
whichlangbenchmark, which measures language choice, not correctness.
Section 08
Field test: submitting the one-line fix to 30 repositories
Section 06 step 02 claims the explicit formatter opt-out is a one-line change worth submitting upstream. On 18 August 2026 that claim was tested against 30 forked repositories carrying explicit ex_doc formatter pins, drawn from this study and a follow-up owner sweep. It largely holds — and it exposes a fourth package state that Section 03 does not separate.
Seventeen of the thirty could take the intended change against their committed dependency state. All seventeen were built, tested and submitted. Four merged inside the first hour.
| Package | Pull request | Status | Verification |
|---|---|---|---|
rebar3_ex_doc | jelly-beam/rebar3_ex_doc#130 | Open | Adds Markdown to the plugin default and to the --formatter help. 19/19 CT, Dialyzer and fixture generation passed. |
phoenix | phoenixframework/phoenix#6794 | Merged in 13m 35s | Merged, then republished docs pushed 4m 18s later. |
credo | rrrene/credo#1308 | Open | Docs emitted Markdown and llms.txt; 1,850 tests and 21 doctests passed. |
oauth2 | ueberauth/oauth2#189 | Open | Docs, formatting, Credo and 55 tests passed. |
guardian | ueberauth/guardian#750 | Open | Preserves HTML and EPUB while adding Markdown; 284 tests passed. |
amqp | pma/amqp#247 | Open | Warnings-as-errors compile and 67 tests against RabbitMQ 3 passed. |
digital_token | ex-money/digital_token#8 | Merged in 36m 52s | Maintainer republished the docs 2m 15s after merge. |
stripity_stripe | beam-community/stripity-stripe#946 | Open | Docs, compile, Credo, Dialyzer and 364 tests passed. |
reactor | ash-project/reactor#337 | Open | mix check --no-retry passed; cites ash as the in-organisation precedent. |
opentelemetry_absinthe | open-telemetry/opentelemetry-erlang-contrib#787 | Open | Docs, compile, format, 33 tests and Dialyzer passed. Blocked on EasyCLA identity, not on code. |
cloak | danielberkompas/cloak#131 | Open | Docs and 71 tests passed. Semaphore fails before checkout on retired machine images. |
configparser_ex | easco/configparser_ex#17 | Open | No committed lock; docs emitted Markdown and 29 tests passed. |
recase | wemake-services/recase#253 | Merged in 53m 37s | Five CI matrix jobs passed; approved, then merged 43 seconds later. |
x509 | voltone/x509#104 | Open | Docs, format and warnings-as-errors compile passed. One expired-certificate failure reproduces on upstream. |
geo | felt/geo#253 | Open | Docs emitted Markdown; 170 tests, 16 properties and 3 doctests passed. |
geo_postgis | felt/geo_postgis#289 | Open | Two PostGIS-version-sensitive failures reproduce unchanged on upstream. |
phoenix_swoosh | swoosh/phoenix_swoosh#499 | Merged in 2m 17s | The fastest response in the campaign. Format and 38 tests passed. |
Merge speed. Phoenix Swoosh merged in 2m 17s, Phoenix in 13m 35s, Digital Token in 36m 52s and Recase in 53m 37s — a median of 25m 14s across the four. No maintainer disputed the technical premise during the window. Two of the remaining thirteen carry red CI that is infrastructure rather than code: an EasyCLA identity check, and a Semaphore config requesting retired machine images.
Small, well-scoped ecosystem maintenance gets reviewed and merged in minutes here. That is a finding about the ecosystem, not a footnote about the campaign.
Merging is not shipping. Of the four merged, only two have republished.
phoenix.hexdocs.pm/llms.txt and digital-token.hexdocs.pm/llms.txt now
return 200; phoenix-swoosh.hexdocs.pm/llms.txt and
recase.hexdocs.pm/llms.txt still return 404. An accepted formatter change reaches
an agent only after the docs are rebuilt and published — Cause B acting on Cause A’s fix.
The fourth state — stale and pinned
The other thirteen were withheld rather than submitted, because the one-line change was
tested rather than assumed and it failed. Their committed ex_doc versions predate 0.40.0, so
adding the formatter alone makes mix docs fail outright.
| State | Safe remediation |
|---|---|
| Current ex_doc, no explicit pin | Already inherits Markdown — publish docs. |
| Current ex_doc + explicit pin | Add “markdown”. Genuinely one line. |
| Old ex_doc, no explicit pin | Refresh ex_doc, republish. |
| Old ex_doc + explicit pin | Refresh ex_doc and add the formatter. Neither alone is sufficient. |
Section 03 splits the causes into an explicit opt-out and a stale build. A repository that commits its lockfile can be both at once, and then neither remedy works alone. 13 of the 30 (43%) were in that fourth state or carried a further docs-configuration incompatibility.
This does not revise Table 2. That census classifies 283 published packages by download volume; this is a 30-repository implementation sample selected for having explicit pins, and the two denominators are not interchangeable. What it revises is the unit of work: from a one-line pull request to a small dependency-aware maintenance change.
| Package | Committed ex_doc | Why the one-line change was withheld |
|---|---|---|
bunt | 0.30.9 | Changed docs cannot load the Markdown formatter. |
nimble_options | 0.35.1 | Markdown worked only after a temporary update to ex_doc 0.40.3. |
open_api_spex | 0.37.3 | CI runs mix docs --warnings-as-errors, which the one-line change fails deterministically. |
logger_json | 0.37.3 | Committed-lock docs fail with formatter module “markdown” not found. |
goth | 0.35.1 | Markdown generation passed only with a temporary ex_doc update. |
sourceror | 0.31.2 | Restored-lock mix check fails when docs try to load Markdown. |
typed_struct | 0.38.4 | Committed-lock docs fail, and contributions target develop rather than main. |
saxy | 0.34.2 | Old ex_doc lacks Markdown; current ex_doc then rejects the existing assets string. |
toml | 0.29.0 | Committed-lock docs fail. |
excellent_migrations | 0.28.4 | Baseline docs already broken: preferred_cli_env selects :docs while ex_doc is dev-only. |
mint_web_socket | 0.39.1 | One version before Markdown support; changed docs fail. |
joken_jwks | 0.36.1 | Docs fail. PR #102 already proposes the ex_doc 0.40.1 lock update. |
ex_twilio | 0.31.2 | Committed-lock docs fail. |
The amended rule. An explicit formatter pin is a one-line fix only when the
documentation build already resolves ex_doc 0.40 or newer. mix.exs permitting a
current ex_doc is not sufficient — mix.lock decides. Verify on the committed
branch with the restored lockfile:
mix deps.get
mix docs
test -f doc/llms.txtRunning mix deps.update ex_doc first proves future compatibility, not that the
branch you are proposing builds.