ARC Prize 2024 Technical Report: What We Learned
AI Summary
The official post-competition analysis of the 2024 ARC Prize results. The top-scoring submission achieved 55.5% by combining a fine-tuned LLM with a learned search procedure — demonstrating that hybrid approaches significantly outperform pure LLM prompting (which peaks around 30%). The analysis explains what technical approaches worked, what didn't, and why the 40-55% score range was particularly hard to penetrate. Chollet draws lessons for ARC-AGI-2 design and for the broader field: the combination of learned representations with explicit search appears more promising than either alone.
Original excerpt
Post-competition analysis. Top score: 55.5% (hybrid LLM + learned search). Pure LLM ceiling: ~30%. Key insight: learned search on top of learned representations is the winning direction.
This is the most important data point for researchers trying to understand what makes ARC hard and how to make progress.
Frequently asked questions
What is "ARC Prize 2024 Technical Report: What We Learned" about?
The official post-competition analysis of the 2024 ARC Prize results. The top-scoring submission achieved 55.5% by combining a fine-tuned LLM with a learned search procedure — demonstrating that hybrid approaches significantly outperform pure LLM prompting (which peaks around 30%). The analysis expl…
Who wrote "ARC Prize 2024 Technical Report: What We Learned"?
"ARC Prize 2024 Technical Report: What We Learned" was written by François Chollet. It is curated in the François Chollet vault on Burn 451, which covers agi evaluation & arc-agi.
How can I read more content from François Chollet?
The complete François Chollet reading list is available at burn451.cloud/vault/francois-chollet. Each article includes an AI-generated summary so you can decide what to read in seconds. Connect the Burn 451 MCP server to Claude or Cursor to query all François Chollet articles as live AI context.
Can I use "ARC Prize 2024 Technical Report: What We Learned" with Claude or Cursor?
Yes. Install the burn-mcp-server npm package and connect it to Claude Desktop, Claude Code, or Cursor. Once connected, your AI can search and reference this article and the full François Chollet vault in real time — no manual copy-paste required.
28 more articles in this vault.
Import the full François Chollet vault to Burn 451 and build your own knowledge base.
Content attributed to the original author (François Chollet). Burn 451 curates publicly available writing as a reading index. For removal requests, contact @hawking520.