Groq: open models at speed, with a free tier.

What is Groq API?

The Groq API is a hosted service for running open-weight language, vision and speech models on Groq's own hardware. You send chat requests in the same format as the OpenAI API and get completions back, streamed if you like, with tool calling and JSON output supported on models that handle them. The distinguishing feature is speed: generation at several hundred tokens per second on mid-sized models, several times faster than typical GPU serving, which makes streaming feel instant and makes multi-step agent loops practical. Groq does not train models; it serves open ones.

How does Groq API work?

Groq designed a processor, the language processing unit, around deterministic, sequential execution with on-chip memory, which suits the token-by-token nature of language model generation better than general-purpose GPUs for latency. Models are compiled to run across many of these chips in a data centre, and the API routes your request to them. From the developer's side it is an ordinary HTTP API: choose a model name from the list, send messages, read the response. The compatibility with the OpenAI format means existing code and libraries work with a base URL and key change.

How to use Groq API

• Create an account on the Groq console and generate an API key.
• Store it in an environment variable; never in code or a browser.
• Install the Groq Python or JavaScript client, or reuse the OpenAI client with Groq's base URL.
• Choose a model from the models list; small Llama or Qwen models for speed, larger ones for quality.
• Send a chat completion with a system and user message; enable streaming for interactive use.
• Watch the rate limit headers in responses and add backoff; log token usage.

How to generate Groq API key

Sign in to the Groq developer console, open the API keys section, click to create a key, give it a name, and copy the value immediately; it is shown once. Put it in an env file that is listed in .gitignore or in your deployment platform's secrets. Create a separate key for each project so you can revoke one without affecting others, and revoke and regenerate at once if a key appears anywhere public. If a key stops working, check that it was not deleted in the console and that your code is reading the right environment variable.

Are Groq API keys free?

Creating a key is free, and the free tier lets you make requests at no cost within rate limits: a number of requests per minute, tokens per minute and requests per day that vary by model and are published on the rate limits page. That tier is enough for learning, prototypes and small personal tools. Paid tiers raise the limits and bill per million tokens at rates that are among the lowest for hosted inference. There is no trick to the free tier; it is the standard developer on-ramp, and it is why I point students here for their first API project.

Why is Groq API free?

The same reason most developer platforms offer free tiers: to let developers try the service, build on it and become paying customers as usage grows. Open models cost Groq nothing in licensing, its hardware is efficient per token, and rate limits keep the free tier's cost bounded. Free-tier usage may also help the provider improve its service; read the terms for what they do with prompts before sending anything sensitive. For a student, the practical takeaway is that the free tier is real and sustainable within its limits, and the upgrade path is straightforward when a project grows.

What is Groq API limit?

Limits are per organisation and per model: requests per minute, requests per day, tokens per minute and tokens per day, with free-tier figures lower than paid ones. Exceeding one returns a rate-limit error with headers telling you when capacity returns; well-behaved clients wait and retry. Context windows are set by the model (tens to hundreds of thousands of tokens). For a product, estimate your peak requests per minute and tokens per minute from logs and compare with the published limits; DocuLens AI batches document pages to stay under them and falls back to another provider if a limit is hit, following the multi-provider pattern I describe elsewhere.

What is Groq API base URL?

Groq exposes an OpenAI-compatible endpoint under its API host with an openai path prefix, documented on its site; set that as the base URL in the OpenAI client and use your Groq key, and chat completions, streaming, tools and JSON mode work as expected. The native Groq client needs no base URL. Keep the URL in configuration rather than hard-coded so switching providers is a one-line change. Check the documentation for the exact current value rather than copying from an old tutorial, since paths and model names are updated over time.

Groq vs Gemini API

Different offerings. Groq serves open models (Llama, Qwen, gpt-oss and others) extremely fast and cheaply; the models are capable but not the frontier, and you pick from a list. Gemini serves Google's own models, including a frontier reasoning model, with native multimodality (video, audio, long documents), a very large context window and a usable free tier. Choose Groq when latency, cost and open models matter, for example high-volume extraction and fast agent loops. Choose Gemini when you need top-tier reasoning or multimodal input. Many of my systems use both: Groq for the fast bulk path, a frontier model for the hard minority.

Does prompt caching reduce latency?

Yes, where the provider supports it. Prompt caching stores the processed state of a repeated prompt prefix, so the time to first token drops because that prefix is not recomputed, and the cached portion is billed at a lower rate. The effect is largest with long stable prefixes such as a big system prompt or a document set. Providers differ in whether caching is automatic or explicit and in what they support; check Groq's current documentation for its caching behaviour. On any provider, the other latency levers are a smaller model, shorter prompts, streaming so users see output early, and a semantic cache in your own app for repeated questions.