Blog

Until recently, autonomous AI evolution was largely considered hypothetical. Recent evidence indicates such evolution is actively emerging in two forms: human-enabled and feral.

Everything in the field of AI seems to change constantly, from models to benchmarks. Yet, some fundamental issues of evaluation remain surprisingly enduring despite all these other changes.

An analysis using the Feedback Forensics app to detect the differences between the Chatbot Arena and the publicly released version of Llama 4 Maverick.

A list of common issues with ranking feedback data relevant both for model evaluation and training.

As we enter the new year, I wanted to share some of my (tentative) predictions about AI development in 2025.

The General AI Assistant (GAIA) benchmark by Mialon et al. (2023) aims to provide a “convenient yet challenging benchmark for AI assistants”. The benchmark consists of 466 questions, each requiring multiple reasoning steps to answer. Many questions require AI systems to use tools (web browser, code interpreter,…) and contain multi-modal input (images, videos, excel sheets,…). Whilst requiring advanced problem-solving capabilities to solve, GAIA’s tasks are simple and cheap to verify with unambiguous (and short) text answers. In this post, I give a short overview of the GAIA benchmark.

A short (non-exhaustive) list of people who put great stuff on the internet and consistently help me learn new things.

The Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2021) is widely used to demonstrate state-of-the-art language model capabilities. Anthropic’s Claude 3, Google’s Gemini and OpenAI’s GPT-4 models were all introduced alongside prominently placed MMLU results. This publicity makes MMLU one of the most prominently discussed benchmarks for language models. Despite the benchmark’s prominence, the exact model capabilities evaluated and evaluation methods are less widely known. In this blog post, I aim to give a short overview of the MMLU benchmark.

There are many excellent explanations and illustrations of the generative pre-trained transformer (GPT) (Radford et al., 2018) and the original transformer architectures (Vaswani et al., 2017). For example, I can highly recommend the write-up by Turner (2023 ) and the video by Karpathy (2023). Nevertheless, I decided to create yet another illustration of GPT for a recent example class I taught. My illustration focuses on two things: (1) Provide a direct connection from a high-level diagram all the way to an actual code implementation of a GPT, and (2) make the illustration as simple as possible (avoiding unnecessary complexity, e.g. by focusing on vector instead of full matrix/tensor operations).

The field of artificial intelligence (AI) is moving fast. Keeping up with the vast number of new research papers on your own is daunting: scrolling through twitter all day does not seem like a good use of time. Below is a number of AI news aggregators that I personally use to get curated updates of AI developments on a regular basis without spending all my time looking at papers: Last week in AI (newsletter/podast) by Andrey Kurenkov, Jacky Liang et al: weekly updates on the latest AI research development. Also available as a podcast. Guide to AI (newsletter) by Nathan Benaich: in-depth monthly analysis of AI developments, from a commercial/startup/venture capital lense. Import AI (newsletter) by Jack Clark: provides a regular stream of very interesting research papers, with detailed summaries.