Judging is a rubric, applied honestly, by a panel.

How does hackathon judging work?

Organisers publish criteria, often with weights: problem and impact, technical implementation, completeness of the demo, design and usability, innovation, and presentation. Judges, usually a panel of three to five from industry and academia, watch each pitch and demo, ask questions, and score each criterion independently on a scale, with notes. Scores are aggregated; the panel discusses teams where judges diverge widely; a ranking emerges, with special prizes for sponsor tracks. Good events brief the judges beforehand, calibrate on the first teams, and give teams written feedback. The judging rubric guide on my answers site shows the one I use.

How to judge a hackathon

• Agree the rubric and weights with the panel before the first team; calibrate on two teams and compare notes.
• Score what you saw: a demo that did not run scores low on completeness however good the idea.
• Ask one question per team that tests understanding: why this approach, what breaks it, what did you measure.
• Separate the pitch from the product; a nervous speaker with a working system beats a polished talker with slides.
• Check for pre-built work and for AI features that add nothing; ask what the model does that rules could not.
• Write two lines of feedback per team; it is the most valuable thing most teams take home.
• Discuss only where scores diverge; do not re-litigate the whole field.

How to become hackathon judge

Organisers invite judges who have built things, can evaluate quickly and are known to a community: engineers and founders, faculty, past winners, sponsor representatives. The path is visibility and reliability: ship work publicly, win or place at events, speak or mentor at community sessions, and tell organisers you are available. Start by mentoring at a hackathon, which puts you in the room, then ask to judge a preliminary round. Judge well, with a rubric and written feedback, and invitations repeat; my own judging started that way and now spans more than ten events. The speaker page on this site explains how organisers can invite me.

Is LLM required for judge?

Two readings. If the question is whether a hackathon judge needs language model expertise: not for every event, but at AI hackathons at least one judge should understand retrieval, evaluation and the common failure modes, or teams with impressive-looking but hollow demos win. If the question is about LLM-as-judge in model evaluation: no, a language model is not required to evaluate outputs. Exact match, schema validation, retrieval metrics and human grading cover many tasks; an LLM judge is useful for free-text quality at scale, and only after it has been checked against human grades on a sample. Both answers come down to the same thing: someone must actually know what good looks like.

Why is model evaluation important in AI projects?

Because without it nobody knows whether the project works. A model produces plausible output on every input, so a demo on three cherry-picked examples proves nothing. Evaluation means a labelled set of real inputs, a scoring rule that matches the task, and a number that moves when quality moves. It lets a team compare two approaches honestly, catch regressions when something changes, state limitations truthfully to a judge or a user, and decide whether the system is good enough to ship. The hackathon teams I score highest have a small test set and a measured result; the ones I remember least had a chatbot and a claim.

Why is model evaluation important in machine learning?

Machine learning models learn from data and can learn the wrong thing: memorising the training set, exploiting a leak, or performing well on average while failing on the cases that matter. Evaluation on held-out data with appropriate metrics is the only way to estimate how the model will behave on inputs it has never seen, to choose between models, to tune thresholds and to detect drift after deployment. It is also how you communicate: an accuracy, a precision and recall, a recall at k, tell a collaborator or a judge what you built in a sentence. Every serious project, from a course assignment to a production system, lives or dies by its evaluation.

What does model evaluation measure?

The gap between what the model does and what it should do, along the dimensions that matter for the use. For classification: accuracy, precision, recall, F1, per class. For retrieval: recall at k, mean reciprocal rank. For generation: faithfulness to sources, relevance, format validity, refusal correctness. For regression: error magnitudes. For any deployed system: latency, cost and failure rate too. A good evaluation reports several, because improving one often degrades another, and includes the failure examples, because the categories of failure tell you what to fix next.

What are model evaluation metrics?

• Classification: accuracy, precision, recall, F1, confusion matrix, ROC and area under it.
• Retrieval and ranking: recall at k, precision at k, mean reciprocal rank, normalised discounted cumulative gain.
• Generation: faithfulness, answer relevance, completeness, exact match or schema validity for structured output, human or judge ratings.
• Regression: mean absolute error, root mean squared error.
• Detection and segmentation: intersection over union, mean average precision.
• Operations: latency percentiles, cost per request, error rate, refusal rate.
• Pick the metric that reflects the real cost of errors in your task; then report it with the sample size.

How to evaluate semantic search

Build a labelled set: real queries, each with the documents or passages that should be returned. Run the search and record ranked results. Compute recall at k (is a relevant item in the top k), mean reciprocal rank (how high the first relevant item sits) and, for graded relevance, normalised discounted cumulative gain. Compare configurations one change at a time: embedding model, chunking, hybrid versus vector-only, re-ranking on or off. Read the failures to see whether they are retrieval misses or labelling errors. Fifty queries show large effects; a few hundred are needed for small ones. This is the retrieval half of the RAG evaluation described elsewhere on this site.

How to create agentic RAG with self evaluation mechanism

Add a verifier step to the agent loop. After retrieval, score whether the evidence answers the question (a cross-encoder relevance score or a model judgement with a rubric) and, if it does not, let the agent rewrite the query or choose another source rather than answering. After generation, check each claim against the retrieved passages (an entailment check or a judge prompt) and, if a claim is unsupported, revise or remove it and say so. Cap the loop at a few attempts and log every step. The confidence gate in RAG.NextUpgrad is the first half of this; the claim check is the second. Evaluate the whole thing on a labelled set, including questions that should be declined, before trusting the self-evaluation.