Overview
Databricks’ practitioner guide covers when and how to build custom LLMs via fine-tuning or pretraining from scratch, targeting practitioners who have already explored prompt engineering and RAG but need deeper customization for domain-specific applications.
Key Frameworks
Three Guiding Principles:
- Start small and work your way up — exhaust simpler techniques (prompting, RAG) before fine-tuning
- Be data-driven — quality training data is the primary determinant of fine-tuned model quality
- Stay practical — focus on measurable improvements, not theoretical model capability
Techniques Spectrum:
- Supervised Fine-Tuning (SFT): Instruction-following and chat specialization on labeled examples
- Continued Pretraining: Extending a model’s knowledge with domain-specific unlabeled text
- Full Pretraining: Training from scratch when no existing model fits the domain — viable at <$50K with modern infrastructure
Data > Architecture: At every stage, the guide emphasizes that data quality, curation, and evaluation drive outcomes more than model architecture choices.
Actionable Takeaways
- Fine-tune when general models cannot match the required quality after RAG and prompt engineering
- Continued pretraining with domain text is the most underused technique — it bridges the gap between general and domain expertise without labeled data
- LoRA (Low-Rank Adaptation) makes fine-tuning accessible by training only a small fraction of parameters
- Build evaluation pipelines before starting fine-tuning — you need to measure improvement rigorously
- Pretraining from scratch is viable for specific domains at <$50K with Databricks/MosaicML infrastructure
Related
- big-book-of-genai — the broader GenAI deployment path this guide details Stages 3-4 of
- ai-engineering-foundation-models — complementary end-to-end framework for AI engineering
- compact-guide-to-rag — RAG as the predecessor technique before fine-tuning