Five bugs in one afternoon: what a real click found that Claude's own checks never did
This is part four of building the live demo for my apidays Australia 2026 workshop with Claude Code. Part one covered the structural flaw in the original FGA model, part two the Vercel Sandbox mistest and the database that snuck in against my own stated constraints, and part three a smaller, more embarrassing pattern of checking a plan against the brief and not noticing I'd already done it twice. This post is about the moment the second half of the demo - the caseworker admin console - got its first real click, and how many things Claude's own testing had missed by the time it did.
The citizen side of the app was solid by this point: sign up, get delegated a case, toggle your AI assistant's access on and off, watch Auth0 FGA allow or deny it live. The build plan's second half was the MCP vignette - a caseworker-facing admin surface managing contractor records, gated by two authorisation layers doing two different jobs: role-based access control decides which tools a caseworker even sees, and FGA decides which specific contractor record they can act on once they're holding a tool. A step-up flow - a fresh, server-asserted approval required just before a sensitive action, on top of whatever access the caseworker already has - sits on top of the riskier action, bulk-disabling several contractors at once.
Claude built the whole thing in one pass: a new relation in the FGA model, tool handlers wired into both a chat route and a real MCP server, a seed script for the mock contractor data, RBAC checks ahead of the FGA checks. It reported the build clean - typecheck passing, lint passing, every route building without error - and asked whether I wanted to test it live or move straight to the next phase. I tested it live. That one click, and the ones that followed it, found five bugs Claude's own checks had every opportunity to catch and didn't.
The first bug: a role that was assigned correctly and shown incorrectly
I signed in as the caseworker test account, went to /admin, and got a clean refusal page telling me I was signed in as a citizen. I wasn't. I told Claude this directly and it went looking - a Management API call confirmed the caseworker role really was assigned in Auth0. The code reading that role, isCaseworker(), also looked correct on inspection. Claude's first answer was, in effect, "both sides look right, I'm not sure why this is happening" - which is a reasonable thing to say, but it's also the point where the actual bug was still sitting undiscovered one layer further down than either of us had looked.
The Auth0 Action that assigns the persona role, which Claude had written and deployed itself two sessions earlier, gated its entire body - including the ID token's custom claim - behind event.stats.logins_count === 1. That's fine for the role assignment itself, no need to redo a Management API write on every login, but setCustomClaim() doesn't persist across logins. It only affects the ID token minted during the specific login flow it runs in. The account's very first login got the claim correctly. Every login after that got an ID token with no persona claim at all, silently defaulting to citizen, with no error surfacing anywhere in the chain - including in whatever testing Claude had done before telling me the feature was ready.
Claude's fix decoupled the two: role assignment stays gated to first login, the claim gets set on every login, sourced from a persisted app_metadata.persona value rather than re-derived from the query string each time. That second detail mattered - Claude's first draft of the fix would have re-derived the persona from the login URL's query parameter on every login, which meant a returning caseworker clicking a bare login link with no parameter attached would get silently flipped back to citizen. I asked what would happen on a plain re-login with no parameter, and that's what surfaced the second problem before it shipped.
The second bug: the fix was correct, and Claude didn't know it wasn't enough
Deployed. Retested. Same refusal. At this point Claude suggested building a debug tool rather than continuing to guess - a small panel at the bottom of every page showing the caller's own raw ID and access tokens, decoded, with a link out to jwt.io. I said yes. It found the answer within the first use: the raw ID token, decoded, had caseworker in it correctly.
session.user - the object the RBAC check was actually reading - didn't.
Same request, same moment, two different answers depending on which field you looked at. I pushed Claude to explain the discrepancy rather than accept it, and that's what sent it into the @auth0/nextjs-auth0 SDK's own source code rather than its documentation. It found that the citizen side of the app calls auth0.getAccessToken() from two routes, and that call can trigger a silent token refresh. The SDK's refresh handler does session.user = tokenSetResponse.idTokenClaims unconditionally, no merge against whatever session.user held before. Auth0's refresh-token grant doesn't necessarily re-run a post-login Action the way an interactive login does, so a refreshed ID token can come back without the custom claim - and the SDK just overwrites the session's own convenience field with it. Visit an unrelated page, silently lose the claim for the rest of the session.
Claude's fix was to stop trusting session.user altogether and decode the session's raw ID token directly on every call instead. The first fix - the Action rewrite - was genuinely correct, and Claude had no way of knowing, from its own testing, that it wasn't sufficient. Every check it had run would have passed on a session that hadn't yet visited the citizen chat and triggered a refresh. The bug was invisible from the code alone, because the code genuinely was right; the defect was several layers down, in SDK internals that had nothing to do with anything Claude had written for this project. Only comparing two fields in the same debug response, side by side, actually surfaced it.
Bugs three and four: the first real prompt against the admin chat
With RBAC finally resolving correctly, I typed "list the contractors" into the admin chat. The model logged a <thinking> line about calling the tool, the tool ran, and then the reply just stopped. No answer, no error.
I'd asked Claude about this exact possibility in an earlier session - whether the chat route's streamText call needed a stopWhen configured - and it hadn't been confirmed either way at the time, just left as an open theory. This time I asked Claude to check it against the actual AI SDK documentation rather than guess again, and the answer was unambiguous: streamText, the AI SDK function that drives both chat routes' model calls, stops after exactly one step unless you configure stopWhen, and if that one step is a tool call, the tool's result never gets fed back to the model for a follow-up response. Both chat routes in the app - the citizen one and the new admin one - had this same gap, because Claude had written the admin route from the citizen route as a template and carried the missing setting over with it. The citizen route had simply never been prompted in a way that reliably triggered the failure, so nobody had caught it there either. Claude added stopWhen: stepCountIs(5) to both - allow up to five steps in a turn, so a tool call and the answer that follows it can both happen before the turn ends - and the model started actually answering.
The <thinking> tag itself had been sitting in the backlog since an earlier session as an unresolved item nobody had chased down. Claude's explanation this time was that it isn't an AI SDK rendering problem or a Bedrock Gateway quirk at all - Amazon Nova Pro, the model behind this chat, has no separate reasoning channel the SDK could intercept and strip. It's trained to emit literal <thinking>...</thinking> text through the same channel as its actual answer, so there's nothing to parse out programmatically. Claude's fix was a plain instruction added to both system prompts telling the model not to do it. I'm treating that as provisional rather than solved - instructing a model not to do something is a much weaker guarantee than a code fix, and one clean test run isn't enough to trust it.
The fifth bug: an authorisation model with nothing to authorise against
With the chat actually answering, bulk-disable came back denied every time, even immediately after a fresh step-up approval. I asked Claude to check whether this was a repeat of an earlier timing bug from the citizen side, where a UI toggle had shown "granted" before the underlying write had actually landed. It wasn't. A direct fga tuple read against the store showed zero tuples existed for the bulk-disable relation, for anyone at all. The seed script Claude had written only ever wrote the ordinary single-action relation - the second relation the model needs specifically for bulk actions had never been seeded by anything.
Claude had built the FGA model correctly and simply forgotten to seed a piece of the data it depends on, which is close to the exact mistake from the very first FGA setup earlier in this project, just recurring at a different layer of the same build. The fix was writing the missing tuples, each referencing the step-up condition by name - the actual approval timestamp gets supplied fresh at check time, never stored on the tuple itself - and I had Claude confirm it directly against the store with a check, rather than trust that the fix alone meant it worked.
A sixth bug: dark mode
Bulk-disable finally worked, and the next thing I told Claude was that the chat bubbles were unreadable with my system in dark mode. Claude found the bubble background was a hardcoded light grey with no text colour of its own, so it inherited whatever the page's foreground colour was - and that flips to near-white under prefers-color-scheme: dark. Light text on light grey. Neither the CSS file nor the chat component had any reference to the other; the bug lived entirely in the gap between two files that had each been written without checking what the other assumed. A pair of explicit light and dark colour classes fixed it.
What actually caught these
None of the five real bugs above showed up in Claude's own type check, lint pass, build, or the direct API calls it ran itself. Every one of those had already come back clean before I typed a single real prompt into the actual chat box. What found them was the same thing every time: a person using the feature the way an actual user would, not the way an automated check would.
Claude has direct access to the Auth0 tenant, the FGA store, the Firestore database, and the deployed app - it can curl any route, run any check, read any tuple, and it did all of those things before calling the work done. None of that substituted for me actually clicking the button, because the failure modes that mattered here weren't "does the endpoint return 200," they were "does the thing that happens after four separate systems interact correctly do what a person would expect." A session refresh silently dropping a claim. A tool call with no follow-up turn. A relation nobody seeded. A CSS variable nobody connected to a Tailwind class. Every one of these needed a human actually using the thing to exist as a bug at all, because every one of them only manifests at the point where a real interaction crosses a boundary between two pieces of code that were each, on their own, correct.
A seventh bug, in the diagram for this very post
Generating the header image above, via gpt-image-1, turned into a small recursive instance of the same problem. The first version clipped two of the five boxes' icons and labels outside their own borders. The second fixed the clipping but dropped a border off one box entirely and drifted the colour scheme. The third got the layout right - five consistent boxes, nothing clipped - and introduced a new defect: the text read "ALL PASSD" instead of "ALL PASSED," a dropped letter in a six-letter word I only caught by actually reading it rather than glancing at the overall shape.
Image models routinely get text wrong inside generated images this way - dropped letters, duplicated letters, malformed words - even when everything else about the composition is correct. It's a well-known limitation, not specific to this model or this prompt, and it's a neat parallel to the whole point of this post: something that looks finished and passes a glance can still fail the moment anyone actually checks it closely. The fourth attempt, with an explicit instruction to double-check every word's spelling, came back correct. That's the image at the top of this post.
Next Steps
The debug token viewer Claude built to chase down the second bug is staying in the app - it turned out to be the fastest way to answer "what does the session actually believe right now" and I'd rather leave it visible than have Claude argue from first principles about SDK internals the next time something looks wrong. The next post in this series will likely be about rehearsing the whole thing at something closer to conference scale, once the admin console has had a proper testing pass rather than the five-bugs-in-one-afternoon version.