Webhooks and async callbacks for AI pipelines: notifying instead of polling

Polling a job-status endpoint every few seconds wastes resources on both sides. A webhook-based callback pattern is often the better fit for slow AI tasks.

Written by

Pranjul Rathour

GenAI Engineer at SCULT INDIA, based in Kanpur, India. Builds production RAG systems, fine-tunes open LLMs and ships agentic AI products; has mentored 200+ college developers. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks.

pranjulrathour41@gmail.com LinkedIn GitHub X Instagram Bluesky Dev.to

Talking through the products he has shipped

A background job for a slow AI task (a long document summary, a batch process) needs the client to find out when it's done. Polling a status endpoint every few seconds works, but wastes requests and adds latency to when the client actually learns of completion.

The webhook alternative

Instead of the client asking repeatedly, the client provides a callback URL upfront, and your system calls that URL once the job finishes. This is a common pattern in APIs designed for this exact scenario.

Best for server-to-server integrations, where the client can run a small endpoint to receive the callback.
Needs retry logic on your side, since the client's endpoint might be briefly unavailable when you try to notify it.
For a browser-based client that can't receive webhooks directly, a lightweight polling interval (every few seconds, not sub-second) or a WebSocket connection is the more practical choice.

See background jobs for slow AI tasks.

— Pranjul Rathour, GenAI Engineer from Kanpur, India. Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at any campus: pranjulrathour41@gmail.com.