TL;DR: A new study rigorously measures the marginal energy cost of running local LLMs on an RTX 3090, finding that cost per million tokens is determined by effective throughput, not just model size.
Summary: The study challenges the assumption that local LLM inference is 'basically free' by conducting controlled benchmarks using real-time GPU power sampling. It measured the marginal energy cost (euros per million output tokens) for various models, from 1B to 106B parameters. The key finding is that cost is driven by the relationship between power consumption and effective wall-clock throughput, not solely model size.
Why it matters: AI builders considering local LLM deployment should prioritize effective throughput and 'thinking time' for agentic models, as these factors significantly impact cost. This research provides crucial data for optimizing local inference setups and making informed cost-benefit comparisons against cloud APIs.
Source: rss