Abstract Large Language Models (LLMs) are transforming how we interact with technology, but their efficient deployment and management can be complex, especially for individual users and smaller teams. This session offers a “down-to-earth” look at how Red Hat Enterprise Linux (RHEL) 10 provides a solid, open-source foundation for bringing these powerful AI capabilities to life.

We will first highlight key advancements in RHEL 10 that directly benefit AI workloads: * RHEL Lightspeed: Discover this AI-powered assistant for Linux administration that acts as a GenAI-powered command-line assistant. It can simplify daily tasks by helping with answering questions, troubleshooting, and deciphering logs using natural language, and even offers AI-powered package recommendations. This feature is designed to bridge the Linux skills gap and make RHEL administration more intuitive. * Image Mode: Learn how this container-native OS deployment approach simplifies building, deploying, and managing the operating system with consistent, container-based updates and rollbacks. This provides a stable and secure environment crucial for running AI applications.

Next, we’ll dive into how RHEL 10 integrates seamlessly with Red Hat AI Inference Server (RHAIS) to serve LLMs efficiently. RHAIS enables you to serve trained models for inference via an API. A core component of this is vLLM, an inference serving engine that offers superior performance through critical optimizations like the PagedAttention algorithm, GPU graphs, compression, KV cache, and continuous batching. We will cover the essential system prerequisites and GPU setup (for both NVIDIA and AMD GPUs) required to get RHAIS (vLLM) up and running on RHEL AI. Attendees will also see how RHEL AI provides OpenAI-compatible APIs for easy application integration.

Whether you’re looking to experiment with LLMs on your Linux desktop or manage inference for a small team, this session will provide practical steps and insights into leveraging RHEL 10, its AI-powered features like Lightspeed, and Red Hat Inference Server to deploy and manage large language models effectively and efficiently, fostering innovation within the open-source community.