TL;DR: A new analysis of 31,352 hourly LLM benchmark scores reveals that between-day performance variation is three times greater than within-day variation, highlighting the instability of production LLM APIs.
Summary: An open-source system, AIStupidLevel, continuously evaluated 49 LLM identifiers from multiple providers across coding, deep reasoning, and tool-calling tasks. The analysis of 31,352 hourly scores showed an average within-day performance variation of 2.8 points, compared to an 8.4-point variation between different days. This indicates that LLM performance from production APIs can fluctuate significantly over longer periods.
Why it matters: AI builders relying on production LLM APIs should account for substantial day-to-day performance shifts in their applications. Implement continuous monitoring and robust error handling to mitigate the impact of these variations on user experience and application reliability.
Source: reddit