Deploying a LLM Chatbot using BigDL-LLM and Llama2 on an Intel Laptop

Due to the ongoing expansion of large language models (LLMs), leading to performance degradation, heightened memory requirements, and increased computational demands, there is a growing urgency for efficient quantization to compress LLMs into a more compact form. (https://stackoverflow.blog/2023/08/23/fitting-ai-models-in-your-pocket-with-quantization/). Additionally, optimization across platforms is pivotal for enhancing the accessibility of LLMs. BigDL-LLM (https://github.com/intel-analytics/BigDL) is designed to make efficient LLM development more accessible all Intel platform users, spanning from CPUs to GPUs, from clients to the cloud.

BigDL-LLM is an open source library designed to run large language models (LLMs) using low-bit optimizations (FP4/INT4/NF4/FP8/INT8) on Intel XPU , for any PyTorch model with very low latency and small memory footprint. BigDL-LLM incorporates a variety of low-bit technologies including llama.cpp, gptq, bitsandbytes, qlora, and more. With bigdl-llm, users can build and run LLM applications for both inference and fine-tuning, using standard PyTorch APIs (e.g., HuggingFace Transformers and LangChain) on Intel platforms. Meanwhile, a wide range of models (such as LLaMA/LLaM2, ChatGLM2/ChatGLM3, Mistral, Falcon, MPT, Dolly/Dolly-v2, Bloom, StarCoder, Whisper, InternLM, Baichuan, QWen, MOSS, etc.) have already been verified and optimized on bigdl-llm.

The presentation will walk the audience the process of optimizing a Llama 2 model unitizing the BigDL-LLM library, and offers a practical session on deploying a chatbot through Llama2 on an Intel laptop. Subsequently, A detailed walkthrough of the material will be covered as part of a broader workshop on LLM Agents. We invite everyone to join us in exploring this exciting journey with the Intel BigDL-LLM.

Speakesr: Jiao Wang, Intel; Guogiong Song, Intel

Attend in person or online (see weblink)