The impact of concurrent requests on LLM inference performance: a case study with Llama-3.1-8B

Large Language Models have become an integral part of modern AI applications, so their performance under different load conditions is critical for practical deployments. While single-request performance is well documented, the behaviour of these models under concurrent load presents unique challenges and trade-offs that warrant detailed investigation.

Our measurements reveal distinct phases in First Token Latency (FTL) behaviour as concurrency increases:

Our throughput analysis reveals complex relationships between concurrency and token generation speed:

Our analysis shows clear trade-offs between different performance metrics:

Our experimental results reveal complex performance characteristics of LLM inference under varying concurrent loads, with significant implications for both system design and deployment strategies. The observed 15.5-fold throughput improvement from single-request to optimal batch performance suggests significant opportunities for system optimisation, but also highlights the inherent trade-offs in LLM serving systems.

The performance patterns we observed suggest that LLM serving systems operate in different efficiency regimes. The initial sharp improvement in system throughput up to 15 concurrent requests, followed by sub-linear growth up to 45 requests, suggests a complex interaction between the GPU's parallel processing capabilities and the memory subsystem. This behaviour is likely due to the GPU's ability to hide latency through parallel execution up to a certain point, after which memory bandwidth and cache efficiency become limiting factors.

The relatively low KV cache hit rate of 26.29% is particularly interesting, as it suggests significant room for optimisation in memory usage patterns. This could be addressed through various mechanisms, such as improved prompt templating or request similarity analysis. However, the high GPU memory utilisation (98.1%) indicates that we're operating close to the limits of the hardware, suggesting that any improvements would have to come from better memory management rather than additional resource allocation.

These results suggest that single-queue architectures may be suboptimal for LLM serving systems. Instead, our results suggest that a multi-tier architecture, with separate serving strategies for different types of workloads, could better serve different use cases. Interactive applications requiring low latency could be served by a low concurrency tier, while batch processing jobs could be routed to a high concurrency tier optimised for throughput.

The sharp performance degradation beyond certain concurrency thresholds suggests the importance of implementing sophisticated queue management systems. Rather than allowing unlimited concurrent requests, systems should implement adaptive admission control based on current load and performance metrics. This could help maintain optimal performance by keeping the system operating in its most efficient range.

Several promising directions for optimisation emerge from our analysis. The relatively low cache hit rate suggests that investigating prompt standardisation and cache warming strategies could yield significant improvements. In addition, the distinct performance regimes we observed suggest that dynamic scaling and load balancing strategies could be highly effective if properly tuned to these characteristics.

The relationship between throughput and latency that we observed also suggests that custom scheduling algorithms designed specifically for LLM workloads could significantly improve system performance. These schedulers could take into account factors such as prompt length, expected response length and cache affinity to make more intelligent scheduling decisions.

While our study provides valuable insights, it also has several limitations that warrant further investigation. Our tests used a specific model size and hardware configuration; understanding how these patterns scale across different model sizes and hardware platforms would be valuable. In addition, while our synthetic workload was designed to be representative, it may not capture all the complexities of real-world usage patterns.

Future work should investigate how these performance characteristics change with different model architectures, different prompt lengths, and more diverse workload patterns. In addition, exploring the impact of different optimisation techniques, such as continuous batching or dynamic tensor parallelism, could provide valuable insights for system optimisation.

The performance patterns we observe also raise interesting questions about the fundamental limits of LLM serving systems, and how close current architectures are to these theoretical limits. Understanding these limits could help guide future hardware and software co-design efforts in this area.

1. Latency critical applications:

2. Batch processing scenarios:

Although not directly derived from our measurements, the following practices can help organisations better manage their LLM deployments:

Future research should examine these patterns across different model sizes, hardware configurations and workload characteristics to develop more generalised optimisation strategies.