Skip to main content
Every call to the inference API answers with a request id. Send a score for that id and it is kept with the request: it shows in the request’s panel under Models → Observability, the page can list only the answers rated good or bad, and the summary above the table shows the share rated good. A score can come from four places, and each keeps its own: your code (the API), an agent (the MCP tool rate_request), the playground and the dashboard. Sending again from the same place replaces what that place sent before.

Endpoint

Use the same API key you call models with. A key made for a coding agent, which can only call models, can send feedback too. Send exactly one of thumbs, rating or score. A send is the whole feedback: a label or comment you leave out the second time is cleared. The answer is the feedback as it was saved:
A key scores the calls of its own environment. A request id that is not the key’s workspace’s and environment’s, or is older than the 30 days requests are kept, answers 404 with the code request_not_found. A call can be scored a moment after its answer has come back. Each key can send up to 600 scores a minute, and a workspace 3,000, apart from model calls; past that the answer is 429.

Score an answer with the OpenAI SDK

The completion’s id is chatcmpl- and the request id, so you can send it as it is. For the other endpoints, read the X-Request-Id header from the raw response.

With plain HTTP

From an agent

The Runcrate MCP server has a rate_request tool, so an agent can score the answers it gets back (an eval loop, a reviewer agent). Its scores are kept as the agent’s, apart from your code’s. Send one of thumbs, rating or score.

In the dashboard

  • Observability: open a request and rate it with the thumbs under Rating, with a note if you like. The panel also lists what your code and your agents sent. The Rating filter lists only answers rated good, rated bad, or rated at all, and the summary shows the share rated good once anything in the selection has a rating.
  • Playground: rate the answer under the result, or any run in Recent runs.
Pressing the chosen thumb again clears your rating.

Reading feedback back

GET https://www.runcrate.ai/api/v1/requests/<request id>/feedback with your API key lists every place’s feedback on a request, newest first.

How scores read

  • Above 0 is a good answer, below 0 a bad one, and 0 (a rating of 3) neither.
  • A request rated in several places counts as rated good if any place rated it good, and as rated bad if any rated it bad.
  • Feedback is kept as long as the request: 30 days.