The Weird Reason Opus Still Writes GPT-3’s code…

Abstract

The biggest weakness in AI-generated code may not be hallucination at all. It is that the frontier models often default to patterns that were overwhelmingly common years ago, even when better architectures, tools, and practices are widely available today. LLMs are trapped by their own training data. The code isn’t hallucinated. That’s what makes it scary...

“You ask a state-of-the-art AI model to help you build anything related to language models or ML engineering”. And it hands you this:

from transformers import AutoModelForCausalLM, AutoTokenizer
# OR possibly with: GPT2Tokenizer, GPT2LMHeadModel

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

It’s 2026 and that snippet is from 2020-2022. And it just came from one of the most capable frontier AI agents and systems ever built like Cursor, Codex, Antigravity, Claude Code, etc.

This is not a small quirk or a “Well, it mostly works” situation. It is a symptom of a systemic failure that quietly degrades the quality of any Vibe-Coded code across the entire field; and it shows up in your model architectures, dependency files, device setup, training optimizers, and even the repository IDs it writes when you explicitly told it a different one. And almost nobody is talking about it...

---

Frequency Bias

Nowadays, almost everyone know about how language models learn (From your teachers to your tech novice friends...)

They don’t internalize what’s Correct or Current. They learn what is Statistically Dominant in their training data.

Between roughly 2019 to 2023, the internet was filled with GPT-2 and EleutherAI GPT-NeoX content like Blog posts, Colab notebooks, Hugging Face tutorials, Medium articles, GitHub repositories / Gists, Stack Overflow answers, YouTube video transcripts. And they were all demonstrating the same pattern. These snippets were copy-pasted, remixed, translated, and re-published across millions of web pages.

EleutherAI companion pattern is equally ubiquitous, so take a look at it:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")
model = AutoModelForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")

By just token frequency, these patterns dominated the pre-training corpus of every frontier model trained in 2023, 2024, and of course 2025. The model does not choose GPT-2 because it thinks it’s the best option. It is doing frequency-weighted pattern matching. And because it saw gpt2 as a string millions of times in the context of “How to do NLP”, that association is structurally encoded at a very deep level of their internals.

Any problem?
Yes! The problem is that the user believes...

“The model choose GPT-2 because it thinks it is the best option. So I shall continue with it without questioning.”

Research from ACM (SE Perspective on LLMs) confirms this directly by saying...

“If an LLM is trained on biased code favoring an older method, it might neglect newer, more secure mechanisms...”

LLM’s training data for code generation often lacks the diversity needed to prevent reliance on obsolete patterns, and if training data favors certain coding methods, then the models consistently generate code aligned with those methods; potentially at the expense of more secure or modern alternatives. The model isn’t confused, it is just doing exactly what it was created for and trained to do, which happens to be wrong to some extent.

---

But What’s Wrong With GPT-2 and GPT-NeoX?

Because “they are old” is not a good technical argument. And I’ll not do that.

1. Architecture: GPT-2 uses a $768$ dimensional n_embd, absolute positional embeddings with a hard n_ctx=1024 context limit, and there is no RoPE, no Grouped Query Attention (GQA), no SwiGLU, no modern attention variants.

   Its architecture reflects the 2019 understanding of what the transformer models are. That is before any rotary embeddings, before Flash Attention, MoE, MoT, before any of the efficiency and capability improvements of modern open-weight models existed. GPT-NeoX-20B, EleutherAI’s 2022 flagship model, was a genuine contribution for its time. But that time was over, around four years ago...

2. Tokenizer: GPT-2’s Byte-Pair Encoding tokenizer has a vocabulary of $50,257$ tokens, built from a 2019 English web crawl dataset. It handles certain things poorly, like multilingual content, and consumes more tokens than necessary for many modern use cases. Every token which is burned on bad tokenization is inference cost and context window you don’t get back...

3. Training alignment: GPT-NeoX was trained on The Pile, which is a 2021 dataset with no RLHF, instruction tuning, DPO, no alignment of any kind. It is just an autoregressive language model whose core function is “Predicting the next token”. Using it as the base for a modern agentic pipeline or fine-tuning experiment isn’t just suboptimal, it’s architecturally incoherent given what we know now.

4. Ecosystem: EleutherAI’s GPT-Neo is already archived and GPT-NeoX has not received any NEW updates. No one is publishing updated evals and no one is fine-tuning them for production use cases. It was a Vibrant community that surrounded these models, which has now moved on. If you start a new project on GPT-NeoX today, you are deliberately inheriting a dead ecosystem, which is unideal.

To visualize just how far the models have moved, take a look at this...

That’s a Six Generations of architectural advancement from GPT-2.

---

Why This Problem Survives

You might assume...

“Well, these models have a late 2025 knowledge cutoff, surely they know about Qwen3 and other latest models?”

And they do (At least the new ones), in a factual sense. If you ask Opus 4.6 or GPT-5.5 to explain what Qwen3 open-weight model is, they’ll give you a answer. But knowing a thing exists and defaulting to it under the Generative Pressure are two very different things.

The difference is about data density, not data recency.

The 2019-2023 period generated more GPT-2 tutorial content than the 2024-2025 period generated Qwen2.5 tutorial content, including chatgpt generated codes with gpt2 / neo. And even with a late 2025 cutoff like November or December 2025, the token-weight from four years of GPT-2 blog posts and Colab notebooks is deeper than the signals from just one year of Qwen tutorials.

This is a data quality problem, not a knowledge cutoff problem. It means the fix is not “Just Wait for the next model release.”

I’d say one of the solutions is to fill the datasets with articles, blogs, posts, conversations that contain human reasoning about such similar general problems which are openly discussed, studied, and well described. Even if they don’t have the solution or fix, having them in data can help model to know certain problems or similar can exist in various tasks like coding, math, writing, etc. Which can possibly improve generalisation and reasoning of the models.

---

Degraded Reasoning and Decisions

The damage is not just limited to a bad and outdated import at the top of a file. Because when a model defaults to GPT-2 as its mental reference implementation for “What a language model looks like”, then the every downstream decision inherits that mindset.

It recommends $768$ dimensional hidden states because that’s what GPT-2 used. It suggests absolute positional embeddings because that’s what GPT-2 used. It reaches for nn.CrossEntropyLoss over a flat vocabulary of 50K tokens because that’s the training setup it saw described the most often and fine-tuning approaches designed for models with no instruction tuning, rather than the approaches designed for the modern aligning methods we know like GRPO and GSPO etc.

If training data is outdated or incomplete, then the models produce incomplete or inaccurate code too. Because of this, the models may unintentionally pick up and spread outdated patterns in their code generation, which will be used by users and probably be released in production or in open-source. And if models are trained on such biased data, then the model reinforces a wrong mental model for user’s request, and that propagates outward.

Effects

Suppose if the agentic-LLMs are trained on such low quality outdated datasets, then they would fail in:

1. Different tokenization: Recognizing difference between tokenizer-based models and tokenizer-free models or token vs byte level difference in models like BLT (Byte Latent Transformer: Patches Scale Better Than Tokens).

2. Different curriculum: Pre-trainining, SFT, RLHF, DPO, etc. are all different stages of model training for transformers or standard Mamba-SSM. It would fail to recognize difference between Models with different architecture and Transformers architecture. The specific change in the curriculum can be the fundamental need of that novel architecture being vibe-coded. And if the agent tries to generalize known patterns, it would fail to even understand that need...

3. Different architectures: Architecture that requires a Tokenizer and architecture that does not. For example, Byte-Latent-Transformer does not require tokenizer, and works purely on Raw Text-Bytes. If someone tries to extend its modality to images, audio, etc. Using such an agentic-LLM, then it would fail to understand that Byte-level processing is efficient for Text due to it’s low size, but get’s utterly inefficient for Image and Audio that can reach hundreds of thousands of Gigabytes. And if the agent generalizes BLT’s method for multi-modal tasks it would be a disaster that cannot even be trained due to current hardware limits...

4. Different training objectives: Loss functions, reward functions, etc. are all different for different type of models. It would fail to differentiate between Objectives of different models with different modalities and for novel architectures with different tasks and aligning methods. Again, that specific change in training objectives can be the fundamental need of that novel architecture being vibe-coded as well. And if the agent tries to generalize known patterns, it would fail to understand that need again.

These are just four Machine Learning related patterns, there exists more and I think you must have experienced one of these or more, like when it came with the most dumbest idea ever, even though it was simple task.

1. The requirements.txt Problem

Ask any AI coding agent to generate a requirements.txt or a pyproject.toml for a PyTorch-based project related to Machine-Learning, and you will almost certainly receive something like this:

# Core deep learning framework
torch==2.2.0
torchvision==0.17.0

# Data handling and numerical computing
numpy==1.26.4
pandas==2.2.1

# Visualization
matplotlib==3.8.3
seaborn==0.13.2

# NLP and deep learning
transformers==4.26.1
datasets==2.14.2
accelerate==0.24.1

NOTE: Above sample snippet was generated by a GPT-5.X series model.

Every single version in that list is a ghost. Like torch==2.2.0 was released in January 2024. And I think how bad dependency conflicts can be needs no explanation.

The reason this happens is the same root cause. Because the dependency configuration files were copy-pasted across millions of tutorials and GitHub repos in 2021-2024, the model’s frequency prior for “What a valid requirements.txt looks like” was built on that corpus. So it reproduces it faithfully.

The damage here is particularly nasty for less experienced developers. Because they run pip install -r requirements.txt, everything installs cleanly, they feel good, and then they hit a wall three layers deep when a function signature was changed between previous and newer versions of a package, deprecated warnings or a CUDA driver doesn’t match the pinned wheel for torch, etc. The error message points nowhere near the actual problem and hours just disappear in waste.

2. The Accelerator Problem

This one is subtle enough that even experienced ones let it slide, which is why I think it is worth naming explicitly.

For years, the orthodox way to handle device placement in PyTorch looked like this:

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

It was in every tutorial, every starter notebook, and every “Getting started with PyTorch” blog post written between 2019 and 2024. The training data of every major language model has it. It’s not wrong, in fact good for normal quick tests/test_*.py scripts and many other cases. However, it is not good enough for main training loops.

The problem here is that it completely ignores Apple Silicon, Intel GPUs, NPUs, etc. If someone runs this code on an Mac Mini (Which is now an extremely common developer machine) then the torch.cuda.is_available() returns False, and the model falls back to CPU, and performance degrades and is super slow. Now we have libraries like accelerate that handles everything from device mapping (including, XPU, NPU and GPU), DDP / FSDP training, CPU / Disk Offloading, saving / loading weights (checkpointing) and optimizer, etc.

3. The Infestation Problem

This one is less about breaking code and more about a deeply Wrong idea of what clean code actually is, and it is everywhere.

Ask any LLM to generate a Python module, a training script, or even a short utility file, and there’s a high chance you’ll get something like this scattered throughout:

# ============================================================
# DATA LOADING SECTION
# ============================================================

def load_dataset(path):
    ...

# ------------------------------------------------------------
# MODEL DEFINITION
# ------------------------------------------------------------

class MyModel(nn.Module):
    ...

# ####################################################
# TRAINING LOOP
# ####################################################

def train():
    ...

# ── Now safe to load ────────────────────────────────────────────────────────
print("Loading model...")

Sometimes it’s # -----, sometimes # =====, sometimes # ##### and the New one is # ── TEXT ──────, or a creative mix of ASCII characters that looks like it escaped from a 1990s C header file. The specific character does not matter. The pattern is the same...

A large decorative comment blocks used as visual section separators inside the source code.

This is pure noise. Real clean code is readable because of good naming, type annotation, logical function boundaries, proper module separation, and clear docstrings; not because of drawing a line of dashes between functions. A well-named function like load_and_validate_dataset() says more than any divider comment ever could.

Where did models learn this pattern?
I think most likely from license headers, enterprise codebases, auto-generated code files, and old-school “style guide” tutorials that were heavy on visual structure for children. Because the model absorbed “section divider = organized code” as a deep association, it now reproduces it as a default definition of “clean code”, which is everywhere.

The cost of this...
First, it’s a waste of tokens, both in generation and in every subsequent context window that has to process that file. A 500-line script with heavy divider comments might be burning 15-20% of its tokens on decorative text-art.

Second, it teaches bad habits to anyone learning from AI-generated code. Developers who see dividers consistently start thinking...

“That’s what professional code looks like...”

The correct standards can be docstrings for classes and functions, type annotation for clarity, logically small files that don’t need internal navigation aids, and nothing else. If a file genuinely needs section markers, it is a sign the file should be refactored into multiple modules.

This one has nothing to do with “modernization” rather the defination of what clean code actually is.

---

Modern Baselines

Model that needs a tokenizer

Qwen2.5, Qwen3, Qwen3-VL, Qwen3.5 / Omni, GLM, MiniMax, Kimi, etc. are the current gold standard for open-source reference implementations in AI/ML code. And there’s strong empirical backing for that claim.

What frontier LLMs still hand you:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model(**inputs)

But this can be the current minimum acceptable baseline for any new AI/ML work that requires a tokenizer:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Qwen/Qwen3-VL-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "She has really beautiful eyes."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Because this reflects that five generations of improvements in tokenization and attention efficiency, context length, alignment and instruction following, and multilingual capability we talked about earlier. The Qwen3 tokenizer alone, with its 150K+ vocabulary, handles code, math, and non-English text in ways that GPT-2’s first generation tokenizer cannot, plus this is a multi-modal model’s tokenizer.

The Optimizer

Ask any agentic LLM to write a training loop for you, any architecture, any task, any amount of total parameters, and with overwhelming probability you will get this on the line of the optimizer block:

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4, weight_decay=0.01)

No question asked. No context evaluated. Just AdamW, default hyperparameters, and move on.

This is the optimizer equivalent of the GPT-2 ID problem. AdamW became the most dominant optimizer around 2018-2019 with BERT and GPT-2, and every Hugging Face tutorial, every paper’s implementation appendix, every “Train your first transformer” Colab notebook for the next four years wrote that exact line. Due to this, the frequency signal got baked in permanently in the models. The following research formally confirmed this as well:

“AdamW has been the default optimizer for transformer pretraining, and for many years the community searched for faster and more stable alternatives with only constrained positive outcomes.” (Cautious Optimizers, 2024).

A separate study on machine learning for software engineering (ML4SE) tasks found that:

“in the majority of the works we consider, researchers used Adam as the default optimizer and did not report any attempts to tune its hyperparameters or choose another optimizer.” (Judging Adam, 2023).

AI agents absorbed this exact pattern and generalized it as...

$$optimizer = AdamW$$

Full stop, regardless of what you’re actually building.

But what’s the problem defaulting to AdamW?

1. Memory: AdamW maintains two optimizer state tensors per parameter, first moment $m_t$ and second moment $v_t$. That’s 2x model parameter count in optimizer state alone. For a 7B model in float32, you’re carrying roughly 56GB of optimizer state before you even touch gradients or activations. AdamW is one of the most memory-hungry optimizers that exists, and AI agents never mention this when generating the training loop.

2. Not-neutral: AdamW was made for transformer NLP tasks. Assumptions about gradient curvature don’t universally hold for SSMs, CNNs, novel hybrid architectures, or anything with fundamentally different gradient landscapes. Defaulting to it for everything is not a neutral choice...

3. The frontier has already moved: This makes the agents’ default particularly embarrassing. While agents are still outputting torch.optim.AdamW(...) by reflex, production-scale labs dropped it. Muon (Momentum Orthogonalized by Newton-Schulz optimizer), is now used to train frontier models at scale, take a look at this:

• Moonlight (16B MoE) was trained with Muon and achieves comparable performance to AdamW-trained counterparts at approximately $52\%$ of the training FLOPs. (Muon is Scalable for LLM Training, 2025).
• Kimi K2 (1 trillion parameters) deployed MuonClip in place of AdamW, reporting dramatically smoother loss curves. (Medium, 2025).
• GLM-4.5 and INTELLECT-3 are both Muon-trained. (PredNext, 2026), (INTELLECT-3: Technical Report).

Keller Jordan, Muon’s creator, described the broad landscape of optimizers as:

“The neural network optimization research literature is mostly filled with a graveyard of dead optimizers that claimed to beat AdamW, something is going wrong with the research. The most common culprit is bad baselines: papers often don’t sufficiently tune their AdamW baseline before comparing to a new optimizer.” (Keller Jordan’s blog, 2024).

That’s what the frontier LLMs like Kimi-K2.x models are using. While AI agents still hand you AdamW(model.parameters(), lr=1e-4).

Not just that, but other optimizers exist too from other different labs:

• Adam-mini (one learning rate per parameter group instead of per parameter, ~half the memory of AdamW). (Adam-mini: Use Fewer Learning Rates To Gain More).

• GaLore (gradient low-rank projection for full-parameter training at fraction of state cost). (GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection).

• Schedule-Free Adam from Meta (no LR scheduler needed, removes one of the most painful hyperparameters to tune). (The Road Less Scheduled).

None of these will appear in AI-generated training loops by default. All of them exist, are well-maintained, and solve real problems.

The Hallucination

This one hits differently because it is not a passive default, it is that the model is actively overriding what you explicitly told it to do. Ask any agentic LLM to write an inference script for Qwen/Qwen3-VL-8B-Instruct (or any latest known model, if you are reading this long after). Spell it out exactly and even with an example. And the model will nod, write the script, and then quietly put this in the code:

model_id = "Qwen/Qwen2.5-7B-Instruct"

OR sometimes even worse, it’ll actually write your repo ID and then add a comment next to it saying (With a double space probably):

model_id = "Qwen/Qwen3-VL-8B-Instruct"  # Note: this model may not be publicly available yet

It’s 2026 and Qwen3-VL is publicly available and has been for some time (Since October 2025). Yet, the model’s memory is so certain that anything newer than its internal reference point does not exist and it argues with you in a comment.
The same issue arises with LLaMA, if you ask for a Llama 4 script, you’ll receive a Llama 3.1 8B repository ID.
If you ask for Qwen3.5, you’ll get Qwen2.5 instead. It quietly defaults to whatever it is most confident about, which happens to be dumb.

This problem is now formally named in the research as: context-memory conflict. A paper from April 2026 specifically looked into this issue for code generation (When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation).

“Wang et al. observed a 25%-38% deprecated API usage rate across eight Python libraries, attributing this failure to both stale parametric knowledge and the absence of real-time API status during inference.”

And critically, even when you provide the correct up-to-date information directly in the prompt...

“LLMs struggle to follow externally provided API update specifications without full documentation, with only $74.64\%$ adoption and $42.55\%$ executable rate under update description alone. Adding API documentation yields the largest improvement, raising adoption to $92.87\%$ and executable rate to $66.36\%$. API modification (P2) remains the most challenging pattern, even though API documentation is provided across both metrics ($87.14\%$ adoption rate and $58.19\%$ adoption rate).

So providing documentation and more useful context works, but not for all cases or in long-term.

Why?
Because of the Parametric knowledge dominance. In simple words, models are trained on static databases, when their internal beliefs clash with what we provided, then they often cannot completely override their prior beliefs, even when instructed to do so. (Taming Knowledge Conflicts in Language Models, 2025).

One study described it as a Dunning-Kruger effect in models:

“the generation process within RAG remains modulated by the generator’s entrenched priors, the parametric knowledge that leads to dogmatic behavior and inefficient utilization of the retrieved context. Often clinging to incorrect internal beliefs rather than factual external evidence.” (What Is Seen Cannot Be Unseen, 2025).

A similar pattern exists for LLMs with tool-usage (Which is even more common now):

“Temporal Discrepancy arises when external tools provide updated information that conflicts with the static, pre-trained knowledge of an LLM to a specific cutoff date.” (Investigating Tool-Memory Conflicts in Tool-Augmented LLMs, 2026).

Even when an agent has web search or a file reader tool returning the correct current repository ID, the model’s parametric prior fights the tool output. And this has a direct consequence for agentic AI engineering workflows. If you’re vibe-coding a new architecture with an AI agent and suppose you specify:

• Use Qwen/Qwen3-VL-8B-Instruct for the vision encoder,
• Use Qwen/Qwen-Image-Edit for the diffusion,

The agent will silently replace all repository IDs with what it is more confident about, like Qwen/Qwen2.5-7B-Instruct, stable-diffusion-v1-5/stable-diffusion-v1-5. Because those are the versions it saw described most frequently during training. The code might look structurally correct. It may even run, but it will be training or inferencing on an entirely different model than the one you specified, without any obvious error.

Any fix?
Yes, Always verify repository IDs on Hugging Face before trusting vibe-coded from_pretrained() calls. For any model released in the last 14 months. This is especially true for multimodal models and reasoning variants (e.g. -Instruct, -Thinking, -Reasoning suffixes), and any model where the version numbers changed in the recent past like -2511 (generally a better fine-tuned version released after a few months), etc.

---

Solutions

Here are a few solutions I suggest.

1. Balanced Data Curation: Labs should explicitly increase the weighting of content containing modern industry standards. Such balanced data ensures the model can learn both recent and outdated code patterns while also deciding default generation behaviors during training.

2. RLHF/DPO Correction: Training the model on human feedback to prefer up-to-date, instead of outdated, references. This gives a stable fix and one that can truly dig into actual model generation behavior rather than patching with system prompts or instructions at inference time. But paradoxically, RLHF is also one of the major reasons why these problems even exist at first place. As of 2026, SKILL.md files have proved remarkably successful in enforcing modern standards by rectifying model behavior.

3. Problem-Aware Curation: This is the most underrated fix, and arguably the most effective one. A model can’t reliably anticipate or avoid a problem if it has never seen a detailed explanation of the issue and has no idea about it. By filling pre-training corpus with high-quality human discussions (not just the solutions, but the reasoning about why something is wrong). This way, the model can build the capacity to recognize anti-patterns even before generating them...

The mechanism is very simple: a model trained on thousands of threads where people argue about Why pinning torch==1.13.0 is not good today, or Why device = "cuda" if ... else "cpu" breaks on multi-GPU setups or different compute hardwares. Due to this, the model will internalize the problem structure itself, not just the surface level fix. That’s fundamentally different from training on a tutorial that shows the correct answer without ever naming what the flaw was.

The best sources for this kind of content are GitHub Issues, Reddit (r/LocalLLaMA, r/MachineLearning, r/learnmachinelearning are where real practitioner frustrations with diverse perspectives is found), Quora and Stack Overflow (question framing teaches the model what confused developers look like and how it should handle), and Medium / Substack (longer-form analysis with analogies, comparisons, and reasoned argument). Research papers that include failure analysis and ablations are equally valuable because they don’t just report what worked, they explain what didn’t and why.

Important thing is that this content does not need to contain the solution to be useful. An issue such as saying “I followed that tutorial and got this error and I have no idea why” followed by twenty replies debugging it, even if unresolved, it teaches the model that this type of problem exists or can, what triggers it, and what the diagnostic process looks like. This generalization capability offers more lasting solutions than the previous ones. Because this is a habit rather than a temporary instruction.

This specfic solution I propose compounds over time like a self-reinforcing approach that becomes a habit. As more people write clearly about these patterns in discussions, in issues, on blogs (like this one you are reading right now); the corpus naturally fills with problem-aware content. Labs that will actively harvest this signal and weight it appropriately in their training pipelines will produce models that catch bad patterns even before generating them, not after.

The failure loop here on the left is self-reinforcing. Because anyone who gets bad code from an AI and does not catch it, and publishes it to a blog or GitHub, adds more stale content for the future training datasets.
The model then picks it up in the next training run, and the cycle goes on. Skill files are currently occupying the space of best practices with thousands of GitHub repositories publishing them, and once they are ripened enough in inference (People using Claude Code, PI, Codex, etc.) and enough data is collected by Labs (Called agent-traces now), they will become eligible for the third solution above with time.

So, “The code isn’t hallucinated. It’s inherited. That’s what makes it scary.”

References

1. SE Perspective on LLMs (ACM Digital Library).
2. Byte Latent Transformer: Patches Scale Better Than Tokens (arXiv).
3. Hugging Face Accelerate Repository (GitHub).
4. Cautious Optimizers, 2024 (arXiv).
5. Judging Adam, 2023 (arXiv).
6. Muon is Scalable for LLM Training, 2025 (arXiv).
7. Going Beyond AdamW: A Practical Guide to the Muon Optimizer (Medium).
8. Want to Accelerate LLM Training? Why Not Try Muon? (PredNext).
9. INTELLECT-3: Technical Report (arXiv).
10. Keller Jordan’s Blog.
11. Adam-mini: Use Fewer Learning Rates To Gain More (arXiv).
12. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (arXiv).
13. The Road Less Scheduled (Meta).
14. When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation (arXiv).
15. Taming Knowledge Conflicts in Language Models, 2025 (arXiv).
16. What Is Seen Cannot Be Unseen, 2025 (arXiv).
17. Investigating Tool-Memory Conflicts in Tool-Augmented LLMs, 2026 (arXiv).