10M-Token Context Windows: What They Mean for Your Whole Codebase

This morning Mirendil introduced Fern, and co-founder and CEO Behnam Neyshabur posted the launch on X: "pretrained from scratch, with up to 10M tokens of context." That's "~0.5M lines of code (roughly Megatron-LM scale) or days of an agent's work, all in context at once," he wrote.

Ten million tokens is a big number. The models I write about here mostly top out around 1 million: Gemini 4 Argon and Kimi K3.1 Pro are both 1M, and Grok 4.7, which I use in my own setup, is 500,000. So the obvious question for anyone with a real codebase: do you just throw the whole repo in and stop worrying about structure? Here's what Mirendil actually claims, what its one long-context test shows, and where I think big context helps and where it doesn't.

From the launch post and the founders' posts on X, all as Mirendil describes it:

• The model: Fern 0.1 is Mirendil-Base after "a short round of post-training." The launch page lists it at 0.1 to 0.5 trillion parameters, taking text and code as input.
• The context: "Fern's architecture, training and serving can support up to 10M tokens, about 23 MB of code, at once," which the company puts at about half a million lines of code.
• What it's for: "high-volume agentic work" for people building AI: auditing training data, debugging a training run from its logs, and reading agent traces end to end. The page also lists working across larger repos and agentic coding.
• Bigger models coming: Fern is the first and smallest of three. Two larger base models are still in pretraining.
• Access: the model card says "Coming soon," and there's an early-access form for "a select few teams and individual researchers." No price, no weights and no public API yet.

Mirendil is new. It formally announced itself in June 2026, and Neyshabur previously co-led a research team at Anthropic. A week before Fern, it published how its base model was built: by one researcher working with agents, in 3.5 months, for less than $10M in compute, by the company's account.

The most useful line came from co-founder and CTO Harsh Mehta, on X: "Fern will also support up to 10M context. We have a lot more work to do on making it as usable as 1M context." That's a lab being straight about the gap between advertised and usable context, and it's the gap this post is about.

Mirendil calls it "a basic test of effective access to the long context" and says "more comprehensive long-context results are coming soon." It ran five tasks from RULER, a standard long-context benchmark, on Mirendil-Base, the base model, not the post-trained Fern 0.1. Its numbers:

• Finding one fact (single needle): 100% at every length from 32K up to 10M tokens.
• Connecting facts (multi-hop questions from HotpotQA): 65% at 1M tokens, 45% at 10M.
• Sample size: each result averages 100 examples, except at 10M, where it's 20.

That's the pattern to expect from any long-context model. Finding a specific string in a huge prompt is the easy part. Reasoning across several pieces scattered through it is harder, and it gets harder as the prompt grows. Mirendil also shows its next-token prediction improving as it reads more of a repository, all the way to 10M. That's a real result, but it's about predicting code, not about answering your question correctly.

• Questions about the whole system. "Where does an order's status get changed?" in an app where the answer is spread across controllers, includes, a cron job and a stored procedure. I said it in my setup post: holding more of the code at once means fewer confident guesses. Fewer, not zero.
• Cross-cutting changes. Renaming a field, changing an auth check or swapping a library touches files you didn't know about. More context means fewer missed spots.
• Long logs and traces. A full error log, an agent's whole run or a day of request logs, read end to end, without you deciding up front which part matters. That's exactly the work Mirendil built Fern for.
• Inherited code with no docs. When nobody knows how it fits together, reading all of it is a reasonable first step.

Advertised isn't usable. The RULER paper tested 17 long-context models. All of them claimed at least 32K tokens, and only half held up well at 32K. Mirendil's CTO is saying the same thing about 10M versus 1M. Treat a context number as a ceiling, not a promise.

Lost in the middle. Researchers found that models often do best when the relevant information is at the beginning or end of the prompt, and noticeably worse when it's buried in the middle, "even for explicitly long-context models." Dump a whole repo in and the one file that matters can end up in the middle.

Cost. You pay for input tokens on every call that sends them. At Kimi K3.1 Pro's listed $2 per million input tokens, one full 1M-token prompt is $2 of input each time ($0.20 if it's cached). An agent that resends it on every step multiplies that. Fern has no published price yet, so nobody can say what 10M will cost.

Speed. The model has to read everything you send before it answers. A huge prompt means a longer wait, and in an agent loop you wait on every step.

What goes in. "The whole repo" usually means .env files, keys, customer exports and old backups too. Everything in the prompt goes to the provider. Search and a tidy repo let you send what's needed and nothing else.

Tests are still the check. More context doesn't tell anyone whether the change works. A test suite does, whatever model you use.

1. Measure your repo. See how big it actually is, and check it against the model you use. Tokenizers differ, so use the provider's token counter. Mirendil's own rough guide is about 23 MB of code for 10M tokens.

git ls-files | xargs cat | wc -c    # bytes of tracked code

2. Keep a one-page map. A short README or ARCHITECTURE file that says where things live and how a request flows. It helps people, it helps retrieval, and it's the first thing a big-context model should read.

3. Search first, then go big. For most tasks, let the agent search and open the right files. Save the full-repo prompt for questions about the whole system.

4. Test it on your own code. Before trusting a big window, ask it a question you know the answer to, where the answer is spread across files deep in the repo.

5. Keep secrets out. Exclude .env files, keys, data dumps and backups from anything an agent can read, whatever the window size.

6. Lean on tests. Have the agent run them, and don't merge what you haven't checked.

7. Wait for Fern's numbers. Until there's access, a price and independent long-context results, 10M is a promising launch, not something to plan a project around.

If someone wrote your software years ago and left, a model that can read all of it at once makes it cheaper to understand what you have, find the risky parts and plan changes. It doesn't make the cleanup optional. The businesses that get the most out of these models will be the ones with code that's organized, documented and tested, because that's what makes any model's answer checkable.

If you've got an app nobody fully understands anymore and want a second opinion on what's in it, get a free quote.

What is Mirendil's Fern?

Fern is Mirendil's first model, introduced October 26, 2026. The company says it's pretrained from scratch and supports up to 10 million tokens of context. Fern 0.1 is Mirendil's base model after a short round of post-training.

How much code fits in 10 million tokens?

Mirendil says about 23 MB of code, or about half a million lines, roughly the size of the Megatron-LM library. The exact number depends on the tokenizer and the code.

Can I use Fern today?

Not generally. Mirendil has an early-access request form for a small number of teams and researchers. It hasn't published pricing or said whether it will release weights.

Is a 10M-token context window as good as 1M?

Not yet, by Mirendil's own account. Its CTO said there's "a lot more work to do" to make 10M as usable as 1M. In the company's RULER test on its base model, finding a single fact scored 100% at 10M, but multi-hop questions dropped from 65% at 1M to 45% at 10M, on 20 examples at 10M.

Do I still need RAG or code search with a huge context window?

Usually, yes. Retrieval keeps prompts small, cheaper and faster, keeps the relevant file from getting lost in the middle, and keeps secrets out. Use the big window for questions about the whole system.

• Mirendil, "Introducing Fern" (Oct 26, 2026)
• Behnam Neyshabur on X, "Introducing Fern, Mirendil's first model" (Oct 26, 2026)
• Harsh Mehta on X, on Fern and 10M context (Oct 26, 2026)
• Mirendil, "Request early access"
• Mirendil, "Pretraining at scale without a pretraining team" (Oct 21, 2026)
• Mirendil, "Announcing Mirendil" (June 24, 2026)
• Hsieh et al., "RULER: What's the Real Context Size of Your Long-Context Language Models?" (arXiv)
• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (arXiv)