44% on ARC-AGI-1 for 67 Cents: The Transformer Trained From Scratch in 90 Minutes on a Single RTX 5090
Hello HaWkers, on September 1, 2026 a post titled "44% on ARC-AGI-1 in 67 cents" hit the top of Hacker News and closed the day with 621 points and 158 comments. The author, Mithil Vakde, who graduated in Engineering Physics from IIT Bombay in 2023, trained a plain 75 million parameter transformer from scratch, in about 90 minutes, on a single rented RTX 5090, and scored 44% on the public evaluation set of ARC-AGI-1. Total compute cost was 67 cents. The same model gets 7% on ARC-AGI-2.
Do you think only people with a GPU cluster can do relevant artificial intelligence research? In this article I show what ARC-AGI is and why it resists LLMs, what Vakde's technical recipe was, how the test-time training behind the result works, where that number sits on the table next to TRM, HRM, o3 and Gemini 3.1 Pro, and how to reproduce all of it for the price of a coffee.
What ARC-AGI Is and Why It Resists LLMs
ARC, short for Abstraction and Reasoning Corpus, was created by François Chollet in 2019 in the paper "On the Measure of Intelligence". Each task is a visual puzzle: you get two or three pairs of colored grids, input and output, and you have to figure out the transformation rule so you can apply it to a new grid. Grids are at most 30 by 30 cells with ten colors. A human solves most of them in minutes, with no training at all.
The beauty of the benchmark is that every task has its own rule. There is no pattern to memorize, and that is exactly why language models, trained to reproduce patterns seen on the internet, struggled with it for so long. ARC Prize, launched in June 2024 with more than 1 million dollars in prizes, turned the dataset into a public competition.
The turning point came in December 2024, when OpenAI presented o3: 75.7% on the semi-private ARC-AGI-1 set at around 20 dollars per task, and 87.5% in the high compute configuration, with cost estimated in the thousands of dollars per task. I told that story in detail in the post about o3 and the new era of reasoning models. Solving ARC-AGI-1 stopped being the question. The question became: at what cost?
That is the context in which 67 cents turns into an interesting number.
What Mithil Vakde Did in 90 Minutes
The core idea of mdlARC, the name of the repository, is simple to describe and hard to execute well. Each input-output pair of a task becomes a sequence of tokens. A small transformer is trained from scratch, autoregressively, on those sequences, and training happens at test time: the model sees the training examples and the inputs of the evaluation tasks, never the answers, and learns everything during the exam itself.
The repository README sums up the math: 75 million parameters, a standard transformer, about 2 hours on an RTX 5090 rented on vast.ai and a total cost of roughly 0.67 dollars. The previous version of the same project scored 27.5% spending 1.80 dollars in under 3 hours on a Google Colab A100. The jump to 44% came from a list of changes that anyone training models today will recognize:
- Modern architecture: SwiGLU instead of GELU, RMSNorm instead of LayerNorm.
- Modest scale: 8 layers instead of 4.
- 3D positional embeddings with RoPE, so the model understands row, column and position within the pair.
- Additive per-task embeddings, which let learning cross between different tasks.
- NorMuon optimizer instead of AdamW.
- Flash attention with variable length training and flex attention kernels.
- Output-only supervision: the model learns to predict only the answer grid, not the input one.
- Fewer data augmentations, which cut the cost and, according to the author, improved sample efficiency.
The training data mixes ARC-AGI-1, the ARC-AGI-2 tasks that do not overlap with the first one, and ConceptARC. The augmentations are color permutations and the eight dihedral symmetries, rotations and mirrors. In the end, the model generates several answers for each task and submits the two most frequent ones, which is the attempt limit ARC allows.
The Technical Recipe: The Grid Becomes a Token Sequence
To understand why a text transformer can solve a visual puzzle, it is worth looking at the tokenization. Each cell becomes a token from 0 to 9, which is the color. Two extra tokens mark the end of the row and the end of the grid, so the model knows where the geometry ends.
# Each grid cell becomes a token from 0 to 9 (the color).
# Two separators mark end of row and end of grid, as in mdlARC.
NEW_ROW = 10
END_GRID = 11
def grid_to_tokens(grid: list[list[int]]) -> list[int]:
"""Flattens an ARC grid into a 1D sequence of tokens."""
tokens = []
for row in grid:
tokens.extend(row) # colors 0-9
tokens.append(NEW_ROW) # explicit line break
tokens.append(END_GRID) # end signal for the model
return tokens
def pair_to_sequence(input_grid, output_grid):
"""Concatenates input and output: the model is supervised on the output only."""
return grid_to_tokens(input_grid) + grid_to_tokens(output_grid)
example = pair_to_sequence([[0, 1], [1, 0]], [[1, 0], [0, 1]])
print(example)
# [0, 1, 10, 1, 0, 10, 11, 1, 0, 10, 0, 1, 10, 11]Notice that the output is not an image, it is a sequence. That makes it possible to reuse everything the industry built for language models, from flash attention to RoPE, on a problem that has nothing to do with language. It was one of the longest debates in the Hacker News thread: is an autoregressive transformer trained on grids an LLM? The author's answer was that a token sequence does not have to be a sequence of words.
Isn't Test-Time Training Cheating?
That was the most repeated question in the discussion, and it deserves a technical answer. The model is trained on the inputs of the evaluation tasks, but it never sees the outputs. The author's analogy: you are born during the exam, you get a training set and you learn everything from scratch while the exam is happening. It is transductive learning, and it is the same family of technique that won ARC Prize in 2024 and 2025.
By the way, test-time training is a cousin of the test-time compute I explained in the post about how o3 scales reasoning at inference time. The difference is that, instead of spending more tokens thinking, the model spends gradients learning from the task examples.
The mechanism that makes this work without leakage is the augmentations. Since each task has only two or three examples, training multiplies those examples by applying transformations that preserve the rule: rotate the grid, mirror it, swap the colors around. If the rule is "paint the largest shape blue", it still holds after a 90 degree rotation with the colors permuted.
import random
import numpy as np
def augment(grid: np.ndarray, seed: int) -> np.ndarray:
"""Generates a variation of the grid that preserves the task rule."""
rng = random.Random(seed)
g = grid.copy()
# 1) Dihedral symmetry: one of the 8 rotation and mirror combinations
g = np.rot90(g, k=rng.randint(0, 3))
if rng.random() < 0.5:
g = np.fliplr(g)
# 2) Color permutation: shuffles colors 1-9 and keeps background 0
colors = list(range(1, 10))
rng.shuffle(colors)
mapping = {0: 0, **{original: new for original, new in zip(range(1, 10), colors)}}
return np.vectorize(mapping.get)(g)
original = np.array([[0, 1, 2], [3, 0, 4]])
print(augment(original, seed=42))The same permutation has to be applied to the input and to the output of the same pair, otherwise the rule breaks. When it is time to answer, the process is reversed: the model generates the output under each augmentation, the code undoes the transformation and the answers that repeat the most win. It is a vote, and it is the reason the model submits the two most frequent grids:
from collections import Counter
def vote_two_answers(candidates: list[tuple]) -> list[tuple]:
"""Takes the outputs generated under several augmentations, already undone
back to the original orientation, and returns the 2 most frequent ones.
ARC accepts 2 attempts per task."""
counts = Counter(candidates)
return [grid for grid, _ in counts.most_common(2)]
# Each candidate is the grid as a tuple of tuples, so it can go into the Counter
candidates = [
((1, 0), (0, 1)),
((1, 0), (0, 1)),
((0, 1), (1, 0)),
((1, 0), (0, 1)),
((1, 1), (0, 0)),
]
print(vote_two_answers(candidates))
# [((1, 0), (0, 1)), ((0, 1), (1, 0))]On the suspicion that the public set is easier than the private one, the author replied in the thread that the same method put him in tenth place on the private Kaggle set. That is not the same as a result verified by ARC Prize, and it is worth saying so clearly, but it is a sign that the number is not a leakage artifact.
Where This Lands on the Table: TRM, HRM, NVARC and Gemini 3.1 Pro
The result ties with two models that became a talking point in 2025 and that rely on recursion to reason. HRM, Hierarchical Reasoning Model, has 27 million parameters and scored 40.3% on ARC-AGI-1. TRM, Tiny Recursive Model, from Samsung's SAIL lab in Montreal, announced 7 million parameters with 45% on ARC-AGI-1 and 8% on ARC-AGI-2. Vakde makes an interesting caveat about TRM: the model is announced as 7M, but the embedding weights trained alongside it go past 100 million, which puts it in a range similar to mdlARC's.
At the opposite end are the frontier LLMs. Gemini 3.1 Pro, in February 2026, reached 77.1% on ARC-AGI-2 at a cost of 0.962 dollars per task in the highest compute mode. And the winner of ARC Prize 2025, the NVARC team from NVIDIA, got to 24.03% on the private ARC-AGI-2 set spending about 0.20 dollars per task, with a 4 billion parameter Qwen and a lot of synthetic data.
A quick script makes the cost comparison honest, because the unit that matters is not the total cost, it is how much each solved task costs:
# Cost per solved task = cost per task / accuracy rate.
# The per-task costs for o3, NVARC and Gemini 3.1 Pro are the ones published by ARC Prize;
# mdlARC's is the total cost divided by the 400 tasks of the public set.
approaches = {
# name: (benchmark, accuracy in %, cost per task in US$)
"mdlARC (Vakde, RTX 5090)": ("ARC-AGI-1 public", 44.0, 0.67 / 400),
"o3 efficient mode (Dec/2024)": ("ARC-AGI-1 semi-private", 75.7, 20.0),
"NVARC (ARC Prize 2025)": ("ARC-AGI-2 private", 24.03, 0.20),
"Gemini 3.1 Pro high (Feb/2026)": ("ARC-AGI-2 semi-private", 77.1, 0.962),
}
for name, (bench, accuracy, cost_per_task) in approaches.items():
cost_per_solved = cost_per_task / (accuracy / 100)
print(f"{name:32} {bench:24} {accuracy:5.1f}% "
f"US$ {cost_per_task:.5f}/task US$ {cost_per_solved:.4f}/solved")What the math shows is that there are two different games. The LLMs play the game of absolute accuracy, and ARC-AGI-1 is solved for them. mdlARC, TRM and HRM play the game of sample efficiency: how much abstract reasoning you can extract from a small model, from scratch, with no pretraining on any internet at all. On ARC-AGI-2, these small models are still in single digits, and Vakde himself acknowledges that ARC-AGI-3, with interactive environments, would require much larger models.
What the Hacker News Thread Discussed
Beyond the cheating debate, three points are worth recording.
The first one is about cost. A commenter warned that the "0.67 dollars" is misleading if someone extrapolates linearly and imagines that 100 dollars would give 65%. The author agreed that the number illustrates efficiency, not linear scale.
The second one is about what LLMs actually learn. For Vakde, frontier models get to ARC through synthetic post-training: they learn to solve ARC tasks, not to reason abstractly in general. It is an old criticism of the benchmark, and it is one of the reasons ARC Prize launched ARC-AGI-3 on March 25, 2026, with interactive environments where the agent has to figure out the goal on its own.
The third one is about the real world. A comment raised the framing problem: the method may be exploiting the limited size of ARC and failing on real distributions. The author partly agreed and left the answer for future work. It is an honest limitation, and it has to go into any excited reading of the result.
How to Reproduce It for 67 Cents
The repository is MIT and the walkthrough fits in a terminal. On vast.ai, an on-demand RTX 5090 goes for around 0.33 dollars an hour in September 2026, which closes the math on the 2 hours. The card has 32 GB of GDDR7, and the author asks for CUDA above 12.8, preferably 13.
# Clones the repository (MIT license); before that, create a venv and install
# torch, numpy, numba, matplotlib and flash-attn
git clone https://github.com/mvakde/mdlARC.git
# Downloads and assembles the datasets: ARC-AGI-1 + ConceptARC + filtered ARC-AGI-2 tasks
cd mdlARC/dataset_building_scripts
python download_and_group.py
python build_datasets.py arc1 --add-conceptarc --with-filtered
cd ..
# Optional: deletes raw data and the solutions file to prove there is no leakage
# rm -r assets_tmp
# rm assets/solutions.json
# rm -r dataset_building_scripts
# Training + inference. Modes: low, medium or high
python run_script.py highThe optional step of deleting the raw data and the solutions file before training exists to prove there is no leakage, and it is worth running if you plan to publish your number. The training script has three modes, low, medium and high, and the author turned off the loss function logging to gain speed.
If you do not want to spend even the 67 cents, the low and medium modes reduce the compute budget and the code runs on a local GPU, more slowly. What you cannot do is run it on CPU in any reasonable time: flash attention is a requirement.
Efficiency Against Scale: What This Changes for Developers
ARC Prize 2026 is underway with more than 2 million dollars in prizes. The ARC-AGI-2 track pays 700 thousand dollars, and the Kaggle rule is harsh: 240 tasks in 12 hours with four L4 GPUs, something like 0.42 dollars per task, with no internet and open source code mandatory. The grand prize requires 85% on the private set. Under that regime, a Gemini 3.1 Pro at 0.962 dollars per task does not even get in the room. It is exactly the regime where a 75 million parameter transformer trained in 2 hours makes sense.
Three practical takeaways for you as a developer.
The first one is that the accessible research frontier is still open. One person, one rented card and an MIT repository tied with Samsung's lab and with the 2025 non-LLM state of the art. If you want to get into AI research, ARC is a field where a 67 cent experiment fits in anyone's budget.
The second one is about the recipe. SwiGLU, RMSNorm, RoPE, flash attention and Muon family optimizers are not exclusive to giant models; they pay off at 75 million parameters just as much as at 75 billion. If you keep a small model in production, a classifier, an embeddings generator, a time series transformer, it is worth reviewing the architecture with that list in hand. I wrote about this movement in the post about small language models and accessible AI, and mdlARC is the most extreme example of it so far.
The third one is about what comes next. Vakde says he is confident that 65% is reachable within the transformer framework and invites anyone who wants to try. ARC-AGI-2 is still in single digits for this family of models, and ARC-AGI-3 is another game. The question the post leaves hanging is the same one from the thread: does the path to general reasoning go through scaling what we already have, or through discovering what makes a small model learn from two examples? In 2026, for the first time, you can test both hypotheses with coffee money.
Let's go! 🦅
📚 Want to Keep Up With What Is Coming?
This article covered the 67 cent transformer that scored 44% on ARC-AGI-1, but the ecosystem changes every week and not everything turns into an article here.
On X I share what I am testing, the behind the scenes of my projects and the news that shows up before it becomes a post.
Follow Me There
💡 Daily content about development, career and the tools I actually use