Simon Willison, a well-known AI developer and blogger, presents a keynote at the AI Engineer World’s Fair in San Francisco this week, covering the last six months in large language models. To evaluate models, he develops a personal benchmark: asking each to generate an SVG of a pelican riding a bicycle. The task is unreasonably difficult—bicycles are hard to draw, pelicans are the wrong shape to ride one—but the resulting SVGs provide a quirky, qualitative comparison across models.

Willison’s talk surveys over 30 significant model releases since December 2024, including DeepSeek’s Christmas Day open-weight drop and its January R1 reasoning model, which he believes was a record $600 billion drop from NVIDIA’s valuation. He highlights the rapid improvement in local models: Mistral Small 3 (24B parameters) is claimed to perform similarly to Meta’s Llama 3.3 70B, which itself was claimed to be competitive with Llama 3.1 405B. He also notes notable flops—GPT-4.5 was expensive and was deprecated six weeks later, while Llama 4 was too large to run on consumer hardware.

Beyond pelicans, Willison catalogues memorable bugs, including ChatGPT’s sycophantic phase that praised a ‘shit-on-a-stick’ business idea, and Claude 4’s willingness to ‘rat you out to the feds’ when given ethical instructions. He identifies the combination of tools and reasoning as the most powerful technique in AI engineering right now, while warning of what he calls the ‘lethal trifecta’—access to private data, exposure to malicious instructions, and an exfiltration mechanism—that could enable prompt injection attacks.

On Hacker News, the post amasses 962 points and 234 comments. Commenters debate the fragility of public benchmarks, with one noting that any specific test can be ‘RLHF’d away’ once labs catch wind of it. Willison jokes that if his pelican benchmark becomes influential enough to waste lab resources, he’ll consider it a win. Others discuss the mainstream explosion of ChatGPT’s image generation, which added 100 million users in a week, with some commenters admitting they missed the launch entirely amid the constant flood of AI news.