The first month of a client AI feature is when you learn whether you designed for cost or for the demo. A user pastes a 300-page manual and asks forty questions; a bug retries a failed call in a loop overnight. These are the controls I put in every product now, and the ones I walk through in shipping workshops.
Cap the context per request
Retrieval returns the top five reranked chunks, not everything that matched. Long documents are summarised hierarchically rather than stuffed. A hard token ceiling per request means the worst case is known before the first user arrives.
Cache what repeats
- Embeddings by content hash — re-uploaded documents cost nothing.
- Retrieval results by normalised query for a short window.
- Full responses for identical questions on identical context, where the product allows.
Route by difficulty
A small or fine-tuned model classifies and handles the frequent, simple requests; the expensive model handles the hard ones it flags. The pattern in small language models after fine-tuning is as much a cost control as a quality one.
Limit what fallback can spend
Multi-provider fallback keeps you up during an outage, and it can also route every request to your most expensive provider for six hours. Cap fallback spend per hour and alert when the cap is hit; see multi-provider LLM fallback.
Kill the loops
- Retries with a maximum count and exponential backoff.
- Idempotency keys so a client retry does not double a call.
- A per-user and per-key rate limit, even for internal tools.
See it
Log tokens and model per request with a feature tag, and build the simplest dashboard: cost per feature per day. The feature that costs the most is rarely the one you expected, and you cannot fix what you cannot see. Students who add this to a portfolio project have a talking point most candidates lack.
— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.