"You can't test AI" is what students say before they ship something that breaks on a Tuesday. You can test most of an LLM application exactly as you test any software, and the non-deterministic part you test statistically. FineTune Studio ships with 107 passing tests; here is how the layers divide.

Test the deterministic parts like normal software

  • Chunking: given this document, these chunks with these boundaries and this heading context.
  • Retrieval plumbing: the query goes to both indexes; results merge with the right fusion.
  • Schema validation: malformed model output is caught and reported, not passed downstream.
  • Fallback logic: a provider error triggers the next provider; a mid-stream error emits a typed event.

Mock the model here. You are testing your code, not the provider.

Golden prompts with expected properties

For the model-facing parts, keep a set of prompts with properties rather than exact strings: the answer cites at least one chunk; the answer contains the number 5; the answer refuses when the context is empty. Run them against the real model in a scheduled job, not on every commit, and alert on failures.

The evaluation set as a test

Your RAG evaluation set is a test with thresholds: recall@5 must not drop below 0.85, false-refusal rate must stay under 5%. Run it before a deploy that touches retrieval or prompts. When it fails, you have caught a regression before a user did.

In CI

  1. Unit and schema tests on every push — fast, mocked.
  2. Golden prompts and the evaluation set on a schedule and before release — slower, real model, budgeted.
  3. A dataset validation step for fine-tuning projects, so a bad file never reaches a GPU.

What students get wrong

Asserting exact model output, which breaks on every model update; and skipping tests for the deterministic 80% because "it's an AI project". The plumbing is where most bugs live, and it is entirely testable. I teach this as the last hour of the build workshop, because a project with tests is the one a recruiter trusts.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.