HTTP 200, business failure: 7 API acceptance checks

An HTTP 200 response only confirms that the server handled the HTTP request. It does not prove that the model produced a complete answer, the JSON is valid, or the intended business action happened. A production-ready LLM integration needs acceptance checks at every layer between the endpoint and the final business result.

The moment an API starts returning 200 OK can feel like the finish line. The response body contains text, three local test runs complete without an exception, and the integration ticket gets marked done.

The real failures usually appear after the feature reaches a live workflow:

• A customer-service bot returns half a sentence.
• A document summary looks complete but silently skips the last two pages.
• The model says it created a support ticket, yet no ticket exists in the system.
• The backend expects JSON and occasionally receives a Markdown code fence instead.
• A stream displays most of an answer, disconnects, and is still saved as successful.
• An automatic retry sends the request three times. The user sees one answer, while usage records show three calls.

Every one of these failures can happen alongside HTTP 200.

A useful launch review should answer seven questions:

1. Did the request reach the intended endpoint and model?
2. Did the model return an answer, a refusal, or an action that still needs to be executed?
3. Did the output finish, or was it cut off?
4. Can the application parse the response reliably?
5. Did the tool-call loop actually complete?
6. Did the streaming response end cleanly?
7. Can you trace this exact request when something goes wrong?

The checks below work for Claude API integrations, OpenAI-compatible endpoints, Dify, n8n, Claude Code, custom agents, and customer-service bots.

Why HTTP success is not business success

An LLM request usually passes through four layers.

Many integration tests cover only the first two layers.

For example, this is a valid HTTP response, but it is not a completed business answer:

{
  "id": "msg_xxx",
  "content": [],
  "stop_reason": "refusal",
  "usage": {
    "input_tokens": 1210,
    "output_tokens": 18
  }
}

{
  "id": "msg_xxx",
  "content": [],
  "stop_reason": "refusal",
  "usage": {
    "input_tokens": 1210,
    "output_tokens": 18
  }
}

Refusals, output-limit stops, and tool requests can all arrive in normal responses. Anthropic’s documentation tells applications to handle states such as end_turn, max_tokens, tool_use, and refusal separately. OpenAI’s Responses API likewise distinguishes completed, incomplete, refusal output, tool calls, and error events.

That is why acceptance testing cannot stop at response.status_code == 200.

Check 1: verify the endpoint, model, and response identity

Start by ruling out a deceptively simple failure: the request succeeded, but it succeeded in the wrong environment or on the wrong model.

Common causes include:

• The Base URL points to staging while the team believes it is testing production.
• A model alias changes and resolves to a different model than expected.
• A fallback model serves the request, but the application does not record that fallback.
• An SDK automatically appends /v1, leaving the final URL with an extra or missing path segment.
• Multiple keys are mixed together, so usage is charged to the wrong project or account.

At minimum, record the following fields for every acceptance run:

environment: staging
base_url: https://gw.apito.ai/v1
requested_model: <model ID currently visible in the console>
returned_model: <model identifier from the response>
request_id: <request ID from the response header or body>
api_key_alias: content-team-staging
test_case_id: basic-text-001

environment: staging
base_url: https://gw.apito.ai/v1
requested_model: <model ID currently visible in the console>
returned_model: <model identifier from the response>
request_id: <request ID from the response header or body>
api_key_alias: content-team-staging
test_case_id: basic-text-001

Do not write a full API key to logs. A purpose-specific alias and the last four characters are enough to identify the configuration without exposing the credential.

When you route requests through apito.ai, use the model IDs currently visible for your account. If an existing application still references the old claudeapi.com domain, update the request address to apito.ai, then rerun the complete regression suite in this guide. Receiving HTTP 200 after the domain change is only the first migration check.

Pass criteria

• The environment, Base URL, and key purpose are unambiguous.
• The requested model matches the expected model, and any fallback is recorded.
• Every test exposes a request ID that support or engineering can use for investigation.
• Logs never contain a plaintext API key.

Check 2: distinguish answers, refusals, and pending actions

A model response is not always ordinary text. Your application should recognize at least three result types.

Answer

The model has finished the current turn. The output can proceed to format and business validation.

Refusal or safety fallback

The endpoint may return a successful HTTP status while the body contains a refusal or a structured refusal item. Do not treat this result as an empty string and retry automatically. That behavior can produce repeated requests without making the task more likely to succeed.

Tool call

The model is asking the application to execute a function, such as looking up an order, creating a ticket, or reading a database. A tool call means the model planned the next step. It does not mean the business action has happened.

A provider adapter can normalize these outcomes into one application-level enum:

from enum import Enum

class ResultType(str, Enum):
    ANSWER = "answer"
    REFUSAL = "refusal"
    TOOL_CALL = "tool_call"
    INCOMPLETE = "incomplete"
    ERROR = "error"

from enum import Enum

class ResultType(str, Enum):
    ANSWER = "answer"
    REFUSAL = "refusal"
    TOOL_CALL = "tool_call"
    INCOMPLETE = "incomplete"
    ERROR = "error"

Keep provider-specific fields out of business logic. The adapter should translate stop_reason, finish_reason, status, and output items into these five states. The rest of the application then works with one stable contract.

Pass criteria

• Refusals are identified separately and produce an appropriate user-facing message.
• Tool calls are never displayed as final text.
• Empty output is interpreted alongside status fields rather than retried indefinitely.
• Unknown states trigger an alert or review instead of defaulting to success.

Check 3: confirm that the answer finished

Grammatically complete text can still be truncated. Common signals include:

• Anthropic-style responses with stop_reason: max_tokens.
• OpenAI-compatible responses with finish_reason: length.
• A Responses API result with status: incomplete and an explanation in incomplete_details.
• A task that requests 10 items but receives only 7.
• JSON without its closing brace.
• Code that ends halfway through a function or Markdown fence.

Character count alone is not enough. Each task needs its own completeness contract.

A small guard can block obvious truncation:

def assert_completion(finish_reason: str | None, text: str) -> None:
    incomplete_reasons = {"length", "max_tokens", "incomplete"}

    if finish_reason in incomplete_reasons:
        raise ValueError(f"Model output is incomplete: {finish_reason}")

    if not text or not text.strip():
        raise ValueError("Model returned empty content")

def assert_completion(finish_reason: str | None, text: str) -> None:
    incomplete_reasons = {"length", "max_tokens", "incomplete"}

    if finish_reason in incomplete_reasons:
        raise ValueError(f"Model output is incomplete: {finish_reason}")

    if not text or not text.strip():
        raise ValueError("Model returned empty content")

This is only a baseline. Production code should combine it with task-specific structural checks.

Pass criteria

• Truncated responses never enter the normal business path.
• The application tells the user that the output is incomplete instead of showing a partial result as final.
• Continuation and retry logic has a hard limit and retains the preceding request ID.
• Long-running tasks have structural completeness checks tailored to the task.

Check 4: validate the format instead of trusting the prompt

Writing “return JSON only” in a prompt does not replace JSON validation.

The model might return:

Here is the JSON you requested:
```json
{"priority":"high","reason":"The customer requested a refund"}
```

Here is the JSON you requested:
```json
{"priority":"high","reason":"The customer requested a refund"}
```

A person can read it, but json.loads() will fail. Other responses may parse correctly while omitting required fields or encoding numbers as strings.

Use three validation gates:

1. Syntax validation: Is this valid JSON?
2. Schema validation: Are the fields, types, enums, and required properties correct?
3. Business validation: Is the amount nonnegative, does the order exist, and is the category in the current business taxonomy?

import json
from jsonschema import validate

TICKET_SCHEMA = {
    "type": "object",
    "required": ["category", "priority", "summary"],
    "properties": {
        "category": {
            "type": "string",
            "enum": ["billing", "technical", "account", "other"]
        },
        "priority": {
            "type": "string",
            "enum": ["low", "medium", "high"]
        },
        "summary": {"type": "string", "minLength": 1}
    },
    "additionalProperties": False
}

def parse_ticket(raw_text: str) -> dict:
    data = json.loads(raw_text)
    validate(instance=data, schema=TICKET_SCHEMA)
    return data

import json
from jsonschema import validate

TICKET_SCHEMA = {
    "type": "object",
    "required": ["category", "priority", "summary"],
    "properties": {
        "category": {
            "type": "string",
            "enum": ["billing", "technical", "account", "other"]
        },
        "priority": {
            "type": "string",
            "enum": ["low", "medium", "high"]
        },
        "summary": {"type": "string", "minLength": 1}
    },
    "additionalProperties": False
}

def parse_ticket(raw_text: str) -> dict:
    data = json.loads(raw_text)
    validate(instance=data, schema=TICKET_SCHEMA)
    return data

If the endpoint supports structured output or JSON Schema, use that capability. Keep application-side validation as well, because the network path, adapters, and downstream code can still introduce malformed or invalid data.

Pass criteria

• Format success over 100 test runs meets the threshold defined by the project.
• Required fields, enums, and types are checked in code.
• Invalid model output is never written directly to the database.
• Repair and retry attempts are capped, and failed samples are retained for analysis.

Check 5: complete the entire tool-call loop

The most dangerous agent failure is treating “the model requested a tool” as “the tool completed successfully.”

A complete tool call has at least five steps:

1. The model returns a tool name and arguments.
2. The application verifies that the tool is allowed.
3. The application validates the arguments and executes the tool.
4. The application sends the tool result back to the model.
5. The model produces a final answer from that result, and the business system confirms its state.

Any step can fail while the original model request still returns HTTP 200.

For example, the model might produce:

{
  "name": "create_refund_ticket",
  "arguments": {
    "order_id": "A1024",
    "reason": "duplicate_charge"
  }
}

{
  "name": "create_refund_ticket",
  "arguments": {
    "order_id": "A1024",
    "reason": "duplicate_charge"
  }
}

This JSON proves only that the model wants to call create_refund_ticket. Whether a ticket exists depends on the execution result and the actual record in the ticketing system.

Your test suite should cover these failure paths:

• A tool name that does not exist.
• A missing required argument.
• An argument with the wrong type.
• A tool timeout.
• A tool that returns 403, 404, 409, or 500.
• The same tool call executing twice.
• A successful tool result followed by an incorrect final answer.
• A high-risk action that proceeds without human confirmation.

Use an idempotency key or business-unique key for writes, refunds, messages, and file deletion. Without one, a network retry can repeat the real-world action.

Pass criteria

• Tool names and arguments are checked against an allowlist and a schema.
• Tool results and the final model answer share the same trace.
• Write operations have idempotency protection.
• High-risk actions require confirmation or a clearly enforced permission boundary.
• The model cannot describe a failed tool action as completed.

Check 6: require an explicit streaming completion signal

Streaming interfaces create a convincing illusion of success: the screen already contains a lot of text, so the task appears complete.

If an SSE connection drops, the frontend may keep every chunk it received. Without a completion event and stop reason, the application can save or display that partial response as a final answer.

Streaming acceptance tests should verify:

• An explicit completion event arrives.
• Chunks remain in the correct order.
• UTF-8 text is reassembled without broken characters.
• Fragmented tool arguments produce valid JSON after reassembly.
• Client cancellation is recorded correctly on both the service and billing paths.
• Midstream disconnects mark the result as incomplete.
• Final usage arrives, or its absence is represented honestly.

OpenAI’s documentation notes that a client can miss the trailing usage data when a stream is interrupted. Anthropic streaming responses likewise expose message, content-block, and stop information across separate event stages. Receiving the first chunk is not success, and a closed connection is not automatically a normal completion.

Represent the state machine explicitly:

idle -> connecting -> streaming -> completed
                            |-> incomplete
                            |-> cancelled
                            |-> failed

idle -> connecting -> streaming -> completed
                            |-> incomplete
                            |-> cancelled
                            |-> failed

Pass criteria

• Only a completion event can move the result to completed.
• Disconnect, cancellation, and failure use different states.
• Tool-argument fragments can be reassembled and validated as JSON.
• Missing final usage is marked as unknown rather than replaced with fabricated token data.
• A user can retry without the application combining an old partial response with the new result.

Check 7: reconcile logs, usage, and errors

The first six checks answer whether a result is usable. The seventh answers whether your team can diagnose it when it is not.

A useful call record should contain at least these fields:

Do not log full keys, government identifiers, phone numbers, raw customer messages, or internal credentials. If an input must be reproduced, store a redacted sample, a content hash, or a short-lived copy protected by appropriate access controls.

If your team uses apito.ai as a unified model gateway, combine the console’s usage and request records with your own trace_id and business status. The platform record tells you what happened during the model call. The business log tells you whether the request eventually created the ticket, wrote the data, or delivered a result to the user. Troubleshooting becomes evidence-based only when those records line up.

Pass criteria

• Every user report can be mapped to a model request ID.
• Request count, retry count, and billed usage can be reconciled.
• Model calls, tool calls, and business writes share one trace.
• Logs are redacted and governed by a retention period and access policy.
• Alerts classify failure types instead of labeling every issue “model error.”

Copyable LLM API launch acceptance checklist

The following checklist can go directly into a PRD, test plan, or launch review.

# LLM API integration acceptance checklist

Project:
Environment: development / staging / production
Owner:
Acceptance date:
Base URL:
Requested model:
Test suite version:

## 1. Endpoint and identity
- [ ] Base URL and environment are correct
- [ ] Requested and returned models are recorded
- [ ] Keys use aliases, with no plaintext key in logs
- [ ] Every request exposes a request ID

## 2. Response semantics
- [ ] Normal answers are recognized
- [ ] Refusals are recognized and explained to the user
- [ ] Tool calls are not treated as final answers
- [ ] Unknown states trigger an alert

## 3. Output completeness
- [ ] length / max_tokens / incomplete is blocked
- [ ] Empty output does not enter the business workflow
- [ ] Lists, sections, and code have task-level completeness checks
- [ ] Continuation and retry attempts are capped

## 4. Format validation
- [ ] JSON syntax validation passes
- [ ] Schema validation passes
- [ ] Business-field validation passes
- [ ] Failed samples are archived

## 5. Tool calls
- [ ] Tool names and arguments use an allowlist
- [ ] Write operations have idempotency protection
- [ ] Tool results and final answers can be correlated
- [ ] High-risk actions require confirmation

## 6. Streaming output
- [ ] Success is recorded only after the completion event
- [ ] Disconnect, cancellation, and failure are handled separately
- [ ] Text and tool-argument fragments can be reassembled correctly
- [ ] Missing final usage is explicitly marked

## 7. Logging and cost
- [ ] trace_id connects model, tool, and business actions
- [ ] request ID can be used for investigation
- [ ] retry_count and tool_call_count are recorded
- [ ] Token usage can be sampled against billing records
- [ ] Redaction, access, and retention policies are confirmed

# LLM API integration acceptance checklist

Project:
Environment: development / staging / production
Owner:
Acceptance date:
Base URL:
Requested model:
Test suite version:

## 1. Endpoint and identity
- [ ] Base URL and environment are correct
- [ ] Requested and returned models are recorded
- [ ] Keys use aliases, with no plaintext key in logs
- [ ] Every request exposes a request ID

## 2. Response semantics
- [ ] Normal answers are recognized
- [ ] Refusals are recognized and explained to the user
- [ ] Tool calls are not treated as final answers
- [ ] Unknown states trigger an alert

## 3. Output completeness
- [ ] length / max_tokens / incomplete is blocked
- [ ] Empty output does not enter the business workflow
- [ ] Lists, sections, and code have task-level completeness checks
- [ ] Continuation and retry attempts are capped

## 4. Format validation
- [ ] JSON syntax validation passes
- [ ] Schema validation passes
- [ ] Business-field validation passes
- [ ] Failed samples are archived

## 5. Tool calls
- [ ] Tool names and arguments use an allowlist
- [ ] Write operations have idempotency protection
- [ ] Tool results and final answers can be correlated
- [ ] High-risk actions require confirmation

## 6. Streaming output
- [ ] Success is recorded only after the completion event
- [ ] Disconnect, cancellation, and failure are handled separately
- [ ] Text and tool-argument fragments can be reassembled correctly
- [ ] Missing final usage is explicitly marked

## 7. Logging and cost
- [ ] trace_id connects model, tool, and business actions
- [ ] request ID can be used for investigation
- [ ] retry_count and tool_call_count are recorded
- [ ] Token usage can be sampled against billing records
- [ ] Redaction, access, and retention policies are confirmed

Do not test only with “hello”

“Hello” can verify connectivity. It cannot validate a business integration.

Prepare at least the following seven test groups, with expected results for each one.

Do not run each case only once. Temperature, routing, load, and model updates can all introduce variation. Repeat core cases, then record success rate, P50/P95 latency, and format-validation rate.

Roll out in stages instead of switching all traffic

Passing acceptance tests does not mean every live request should move to the new path at once.

A safer rollout sequence is:

1. Shadow test: Send a redacted copy of requests to the new path without returning its output to users.
2. Internal release: Enable it only for employees or test accounts.
3. Low-volume canary: Route by user, project, or percentage.
4. Traffic expansion: Watch error rate, format failures, tool failures, and cost.
5. Fallback retained: Switch back to the old path when a predefined threshold is crossed.

Write rollback conditions as numbers. For example, stop expansion if format-validation rate falls below 99%, tool-call success falls below 98%, P95 latency exceeds the business limit, or per-task cost exceeds the budget. These are examples, not universal defaults; set thresholds according to the risk of your own workflow.

Conclusion

HTTP 200 is a network-layer receipt. The user needs a complete answer, correct data, and a business action that actually happened.

Between those two points sit state classification, completeness checks, format validation, tool execution, streaming completion, and traceable logs. Once these seven checks become part of the launch review, vague reports such as “the model sometimes fails” turn into test records that can be reproduced, assigned, and fixed.

FAQ

What does HTTP 200 actually mean?

It means the server accepted the HTTP request and returned a response. It does not guarantee that the model answer is complete, the format is valid, a tool executed, or a database write succeeded.

The API returned 200 with empty content. Should I retry?

Not immediately. Check stop_reason, finish_reason, status, and the output items first. Empty content can represent a refusal, tool call, incomplete output, or endpoint error. Retry only after classifying the failure as retryable, and enforce a retry limit.

Is finish_reason: length or stop_reason: max_tokens a success?

The network request succeeded, but the output did not finish. Mark it as incomplete. Depending on the task, you can raise the output limit, reduce the input, continue generation, or ask the user to run it again.

If the prompt says “JSON only,” do I still need a schema?

Yes. A prompt is a generation constraint; a schema is an application acceptance check. The model can still add explanatory text, omit fields, or return the wrong type. Validate the output before writing it to a database or triggering a tool.

Does a tool call mean the task is complete?

No. A tool call is a requested action. The application still needs to validate the arguments, execute the tool, process the result, and verify the real business state. Writes and payment-related actions also need idempotency and appropriate confirmation.

Why wait for a completion event after the stream shows text?

A connection can fail after any chunk. Without the completion event and stop reason, the application cannot tell whether the response is final or partial. It may also miss the final usage data.

What should I test first after connecting to apito.ai?

Confirm the Base URL, API key, and models currently visible to the account. Then test a normal answer, truncation, JSON output, tool calls, a streaming interruption, and traceable logs. If you changed an existing application from claudeapi.com to apito.ai, rerun the same regression suite after the address update.

Sources

• Anthropic: Handling stop reasons
• Anthropic: Errors
• Anthropic: Streaming Messages
• OpenAI: Streaming events
• OpenAI: Function calling
• OpenAI: Structured Outputs

Next steps

If you are migrating model endpoints or standardizing model access across a team, review the models and integration settings currently available in your apito.ai account, then run this checklist before moving production traffic.