Tech Beetle briefing US AI

The AI compute gap: Enterprises are buying infrastructure faster than they can measure what it costs

Essential brief

A recent survey of 107 enterprises reveals that AI infrastructure spending is accelerating faster than organizations can measure or control its costs. While most currently rely on hyperscalers and

Key topics

ai compute enterprises buying infrastructure faster than they buying infrastructure faster than they measure costs AI

Key facts

Enterprises are accelerating AI infrastructure spending faster than they can measure costs or utilization.

Most AI workloads currently run on hyperscalers and model-provider APIs, but many plan to evaluate specialized AI clouds soon.

GPU utilization is low, with 83% of enterprises using less than half their capacity, indicating inefficiencies.

Cost tracking is limited; fewer than half of enterprises rigorously monitor AI compute expenses and returns.

Highlights

Only 21% of enterprises run AI at scale in production; most are still experimenting or partially deployed.

Google Cloud leads current AI infrastructure usage at 48%, with hyperscalers and major model APIs dominating.

45% of enterprises plan to evaluate AI-specialized clouds within 12 months, despite minimal current use.

83% report GPU utilization at 50% or less; nearly half operate at 25% or below capacity.

64% intend to switch or add AI infrastructure providers within a year; 38% plan changes within a quarter.

Why it matters

The growing compute gap in enterprise AI infrastructure spending underscores a critical challenge: organizations are investing heavily in AI capabilities without sufficient visibility into the associated costs and utilization. This disconnect can lead to inefficient resource use, increased expenses, and difficulties in scaling AI deployments effectively. Addressing this gap is essential for enterprises to optimize their AI investments and maintain competitive advantage as AI adoption expands.

A survey of 107 enterprises with over 100 employees shows that AI infrastructure spending is increasing rapidly, but organizations struggle to accurately measure and manage the associated costs. Currently, most enterprises run AI workloads on established hyperscalers like Google Cloud, Microsoft Azure, and AWS, as well as major model-provider APIs. However, nearly half of the respondents plan to evaluate specialized AI clouds and alternative accelerators within the next 12 months, despite minimal current usage of these platforms.

Only 21% of surveyed enterprises report running AI at scale in production, with the majority still in experimental or partial production phases. This early stage of deployment means that infrastructure footprints and costs are expected to grow significantly. Despite this, GPU utilization rates remain low, with 83% of enterprises reporting usage at 50% capacity or less, and nearly half operating at 25% or below. This underutilization contributes to inefficiencies and increased costs.

Cost visibility is limited, as fewer than half of enterprises rigorously track the expenses and returns of their AI compute investments. While integration with existing technology stacks and total cost of ownership are the primary factors influencing provider selection, only 8% prioritize headline pricing such as cost per million tokens. This disconnect between spending priorities and measurement capabilities creates challenges in optimizing AI infrastructure investments.

Additionally, 64% of enterprises intend to switch or add AI infrastructure providers within the next year, with 38% planning changes within the next quarter. Most switching interest remains focused on incumbent providers rather than new entrants. Enterprises also show limited awareness of emerging constraints, such as the shift from GPU compute to memory bandwidth as a bottleneck in large-scale AI inference.

Overall, the findings indicate a widening compute gap where rapid investment in AI infrastructure outpaces the ability to monitor and control costs effectively. This gap poses risks for efficient resource allocation and may affect the scalability and sustainability of enterprise AI deployments in the near future.

Key topics in this update include ai compute, enterprises, and buying infrastructure faster than they.