AI agent glossary
Batch API
A way to submit many model requests at once for asynchronous processing, trading immediate responses for lower prices and higher throughput.
Not every model call needs an answer in seconds. A batch API lets you upload a large set of requests, such as thousands of documents to classify, and collect the results when they are ready. Anthropic offers this as the Message Batches API, and OpenAI has a similar Batch API.
Batches are processed asynchronously, so results may take minutes or hours, up to a deadline set by the provider. In exchange, batch requests are billed at a significant discount compared with standard calls; check the current pricing page for the exact rate.
Batching suits evals, backfills, nightly reports, and bulk data processing. It does not fit an interactive agent loop, where each step depends on the previous result, but it can be ideal for running your eval set or for a step that processes many independent items.
Each night, a team submits a batch asking a model to tag the day's 5,000 support tickets by topic, then loads the results into a dashboard the next morning.
Learn more in the course
Related terms
Go from definitions to doing
Learn to use and build AI agents in 65 practical lessons.