One of the most common questions in applied AI development is: should I fine-tune a model, or should I just write better prompts? The answer depends on your specific requirements, budget, and timeline — and getting it wrong costs significant time and money. This guide gives you a clear decision framework based on real-world evidence.
Understanding the Two Approaches
Prompt engineering means crafting the text inputs (system prompts, few-shot examples, chain-of-thought instructions) that guide the model's behaviour without modifying its weights. The model itself remains unchanged — you're just giving it better instructions at runtime.
Fine-tuning means training the model on a custom dataset that updates its weights. The result is a specialised model that has internalised new knowledge or behaviour patterns. Fine-tuned models are invoked like any other API — you just call a custom model endpoint instead of the base model.
When Prompt Engineering Is Enough (Usually)
The answer for most teams, most of the time, is: start with prompt engineering. Modern frontier models like Claude Fable 5, GPT-5.6 Terra, and Gemini 3.1 Pro are extraordinarily capable out of the box. With good prompt engineering, you can achieve surprisingly high performance on specialised tasks without any training.
Prompt engineering wins when:
- Your task is well-described in natural language and doesn't require proprietary knowledge.
- You need to ship quickly — a well-engineered prompt can be deployed in hours; fine-tuning takes days to weeks.
- Your use case evolves frequently — updating a prompt is trivial; updating a fine-tuned model requires a full retraining cycle.
- You have fewer than ~50 high-quality examples — this is typically insufficient for effective fine-tuning.
- You're working with GPT-5.6 Terra or Claude Fable 5, which follow complex instructions exceptionally well, reducing the need for fine-tuning to achieve behavioural customisation.
When Fine-Tuning Genuinely Helps
Fine-tuning has a clear set of use cases where it dramatically outperforms prompting:
1. Proprietary Knowledge and Domain Vocabulary
If your task involves specialised terminology, proprietary document formats, or internal jargon that isn't well-represented in training data, fine-tuning on your internal data can dramatically improve accuracy. Legal contract parsing using your firm's specific clause patterns, or medical note extraction using your hospital's structured note formats, are prime examples.
2. Consistent Output Formatting
If you need 100% consistent output formatting (e.g., every response must be valid JSON matching a specific schema, or every code snippet must follow your exact style guide), fine-tuning enforces this at the weight level rather than relying on in-context instructions that can occasionally be ignored.
3. Latency and Cost Optimisation
Fine-tuning a smaller model (e.g., GPT-4o Mini or Mistral Small 3.1) on your specific task can produce quality matching a much larger model at a fraction of the inference cost and latency. If you're running millions of API calls per month on a narrow, well-defined task, this is often the economically optimal path.
4. Persona and Tone Consistency
For products that require a very specific, consistent voice — a branded AI assistant, a customer service bot with a defined personality — fine-tuning gives you tighter behavioural control than system prompts alone.
The Decision Framework
Use this step-by-step framework to decide:
- Can you describe the task fully in a system prompt? If yes, start with prompting. If the task requires knowledge the model doesn't have, move to step 2.
- Do you have 500+ high-quality, labelled examples? Fine-tuning requires substantial data. Fewer than 100 examples rarely produces meaningful improvements. If you don't have enough data yet, collect it first.
- Is your task stable and well-defined? If your task definition changes frequently, prompting is lower-maintenance. Fine-tuning is best for stable, production tasks.
- What's your performance gap? Measure your current prompt-based performance first. If you're at 82% accuracy and need 95%, fine-tuning is likely necessary. If you're at 76% and need 80%, better prompt engineering is almost certainly the faster path.
- What's your cost/latency constraint? If you need sub-100ms responses or you're processing 100M+ tokens/month, fine-tuning a smaller model is worth the engineering investment.
Practical Costs of Fine-Tuning in 2026
OpenAI's fine-tuning for GPT-4o Mini costs approximately $0.003 per 1K training tokens. A typical fine-tuning run on 10,000 examples (averaging 500 tokens each) costs around $15 — extremely affordable. The real cost is data preparation and iteration time, not compute. Expect 2–6 weeks of engineering time to build a production-quality fine-tuning pipeline, including data cleaning, validation, and evaluation infrastructure.
For open-source fine-tuning (fine-tuning Llama 4 or Mistral on your own GPUs), the story is different: the compute cost is lower per run but the infrastructure investment and operational overhead is significantly higher. This path makes sense for teams with existing ML infrastructure and data privacy requirements that prevent sending training data to third-party APIs.
The Hybrid Approach
Many production AI systems use both: a fine-tuned smaller model handles the high-volume, routine cases cheaply and quickly; a frontier model (accessed via prompt engineering) handles the complex edge cases. This hybrid routing approach — sending 90% of traffic to a cheap fine-tuned model and 10% to an expensive frontier model — often delivers the best quality-to-cost ratio at scale.
Bottom Line
Start with prompt engineering. Measure your performance carefully. Only invest in fine-tuning when you have (1) a stable, well-defined task, (2) sufficient high-quality data, and (3) a demonstrated performance gap that prompt engineering alone cannot close. For the vast majority of AI product use cases in 2026, a well-crafted system prompt with few-shot examples will get you 90% of the way there in 10% of the time.