Choosing Your Base Model
The open-source LLM landscape has exploded. In 2026, you have genuinely capable models at every size point. Smaller models from 7B to 13B parameters run on a single GPU and are fast to fine-tune. Larger models at 70B parameters or more require multi-GPU setups but produce higher-quality outputs. Meta Llama 3 family has become the foundation for a huge proportion of enterprise fine-tuning projects. Llama 3.1 70B delivers performance competitive with much larger closed models and Llama 3.1 8B is an excellent starting point for domain adaptation. Mistral AI models offer excellent performance-per-parameter ratios and are popular for enterprises where inference cost is a primary concern.
Preparing Your Training Data
The quality of your fine-tuning data is the most important factor in the quality of your fine-tuned model. For instruction fine-tuning you need data in a question-answer or instruction-response format. Sources include customer support ticket histories sanitized to remove PII, product documentation and knowledge base articles, internal FAQ documents, and expert-written responses to common questions. For domain adaptation, 1,000 to 10,000 high-quality examples can produce meaningful improvement over the base model. A few hundred excellent examples will outperform thousands of mediocre ones. Have domain experts review and curate a set of gold-standard examples.
Fine-Tuning with LoRA and QLoRA
LoRA (Low-Rank Adaptation) adds a small number of trainable parameters to the model without changing the original weights. Instead of updating all 7 billion parameters during training, LoRA learns a compact set of adapters that add less than 1 percent of the original parameter count. QLoRA combines LoRA with quantization, reducing weight precision from 16-bit floats to 4-bit integers to dramatically reduce memory requirements. A 7B parameter model that would normally require 28GB of GPU memory can be fine-tuned on a single 24GB consumer GPU using QLoRA. The Hugging Face PEFT library provides excellent implementations and the trl library provides the SFTTrainer class that handles the training loop with minimal boilerplate.
Evaluating Your Fine-Tuned Model
Build a holdout evaluation set that was never used in training covering the range of use cases your model needs to handle. Evaluate both the fine-tuned model and the base model using human raters or automated metrics like BERTScore, ROUGE, or exact match. Red-teaming is equally important: systematically try to make the model produce incorrect, harmful, or embarrassing outputs. This is especially important for customer-facing applications where model failures are visible to users.
Deploying for Production Inference
vLLM is the leading open-source LLM inference library implementing PagedAttention to dramatically increase inference throughput. It exposes an OpenAI-compatible API, making it a drop-in replacement for OpenAI’s API in your applications. For cloud deployment, AWS SageMaker, Google Vertex AI, and Azure ML all support custom model deployment with auto-scaling. A production deployment includes health checks, monitoring of response latency and throughput, logging of all requests and responses, and rate limiting to prevent abuse.
