We failed our first OpenAI App review. Not on branding, not on policy — on the integration contract itself. Here's exactly what broke, why, and what we changed about how we ship agent-facing surfaces since. This describes the review process as we experienced it in 2026 — review criteria evolve, so treat the specific mechanics here as a snapshot, not a permanent spec.
Engineering TailyX — Part 4 of 4
Previous: Why We Built an AI Qualification Engine Instead of Another Chatbot
Going in, we assumed the review would mostly be a content and policy check — does the app do what it says, is it safe, does it violate usage terms. Some of that happened. The bulk of the actual back-and-forth was a functional integration test: the reviewer's tooling calls your endpoints exactly the way a real agent would, and it either works or it doesn't. There's no partial credit for "the concept is right but the payload shape is wrong."
Our first submission failed on a mismatch that's embarrassingly simple to describe and easy to introduce in any Rails app: our controller expected qualification answers under one parameter name, while the schema we'd published — the one the review tooling built its request from — declared a different name, scoring_answers versus params[:answers]. The review tooling did exactly what it should: built a request from our own published contract, sent it, and got back a response that didn't match what the schema promised. Correctly failed, for exactly the right reason.
The schema you publish isn't documentation of your API — it is your API, as far as any agent calling it is concerned. If the two drift, the agent's version wins the argument every time, because it never sees your controller code, only the schema.
The fix itself was small. The lesson is bigger: this bug class is invisible in normal human QA. A person testing the widget in a browser never hits the schema-to-controller boundary, because a human doesn't read your OpenAPI spec before clicking a button. An agent — or review tooling acting like one — hits that boundary on the first request.
A related failure mode: treating a tool's output schema as documentation rather than a contract, on the assumption your own code obviously returns something reasonable. Review tooling — and production agents afterward — validate the actual response against the declared outputSchema, field by field. Extra undeclared fields, a looser type than declared, or a field that's sometimes present and sometimes null without that being reflected in the schema all produce failures that look mysterious until you diff the schema against a real response payload line by line.
Tool annotations — read-only or not, safe to call speculatively or not, has side effects or not — aren't decoration. They inform how aggressively an agent is willing to call a tool without explicit user confirmation. Under-annotating a state-changing tool as read-only is the kind of mistake that either gets caught in review or, worse, doesn't, and causes a real production incident the first time an agent calls it more liberally than intended.
outputSchema in CI, not just at write-time.Practically: expect at least one rejection cycle even with a correct integration, budget real calendar time for the round trip, and treat the first rejection as free QA rather than a setback. The reviewer's tooling found, in minutes, the exact bug class that would otherwise have surfaced in production the first time a real user's agent tried to submit a lead — just with a live user watching it fail instead of a review dashboard.