A green suite over a broken feature

I went to walk the landlord onboarding flow end to end, as a customer would: sign in as the platform operator, register a landlord, create their admin account. The first step failed.

Missing CSRF token

At that moment the backend suite was 169 tests green across 26 files, the frontend suite was green, and the feature had been "working" for months in development. Nobody had ever created a landlord through the production UI, because the only people who use that screen are me and the demo, and the demo accounts were seeded directly into the database.

The interesting part is not that there was a bug. It is that no test I could have written in the normal way would have caught it, because the code was correct. What was wrong was where the code was running.

The hypothesis I started with, and why it was wrong

My first guess was that the token lived in Redux and a full page load had wiped it. I had typed the URL directly instead of clicking through the app, so that fit.

It was wrong. I logged in again and navigated only by clicking links, so nothing reloaded. Same error.

That is worth saying out loud, because the wrong hypothesis was plausible and I would have "fixed" something that was never broken. The thing that killed it was not more thinking. It was reproducing the failure under conditions that excluded the theory.

What was actually happening

PropertyOS uses the double-submit pattern for CSRF. The server sets a random token in a cookie, and the client has to read that cookie and echo the value back in a header. An attacker's page can cause the browser to send the cookie, but cannot read it, so it cannot construct the matching header.

That pattern has one hard requirement: the frontend must be able to read the cookie. So the cookie is deliberately not httpOnly:

export function csrfCookieOptions(env) {
  const isProd = env.nodeEnv === "production";
  return {
    httpOnly: false,
    secure: isProd,
    sameSite: isProd ? "none" : "lax",
    maxAge: 7 * 24 * 60 * 60 * 1000
  };
}

That is correct. httpOnly: false is the whole point.

Here is what I had not thought through. In production the two halves of PropertyOS live on different hosts:

  • the frontend on mypropertyos.de
  • the API on api.mypropertyos.de

The cookie is set by a response from the API, so it belongs to api.mypropertyos.de. I confirmed this rather than assuming it, by logging in with curl and reading the cookie jar:

api.mypropertyos.de  FALSE  /  TRUE  …  csrf_token  662582ce338a…

httpOnly: false means "JavaScript may read this cookie" — but only JavaScript running on api.mypropertyos.de. Nothing runs there. The frontend, on a different host, sees nothing at all. I checked that too, in the browser:

document.cookie          // ""
localStorage             // empty
sessionStorage           // empty

So the frontend helper that built the header:

headers: { "x-csrf-token": getCookie("csrf_token") }

was resolving to undefined on every single call, and the server was correctly rejecting every mutation.

The refresh token still worked, which is why the session looked healthy and nothing else seemed wrong. The browser sends cookies back to the host that set them automatically. Sending is not the problem. Reading is.

Why development could never show me this

In development both halves run on localhost, on different ports. Ports do not scope cookies. localhost:5173 and localhost:6060 are the same host as far as the cookie jar is concerned, so the frontend reads the cookie, the header goes out, and everything works.

The bug is not in any line of code. It is in the relationship between two deployment targets. A unit test runs in one process. An integration test runs against one server. Neither can express "these two things are served from different registrable domains," because in the test environment they are not.

This is the real lesson, and it is uncomfortable: a test suite verifies that the code is right. It cannot verify that the deployment is right. The only thing that tells you about production is production.

Half the app worked, which made it look like something else

The confusing part was that the landlord admin side of PropertyOS was completely fine. Properties and units could be created all day.

Because those were written differently. They take the token from Redux, populated from the login response body, and pass it in:

const csrfToken = useSelector((state) => state.landlordadminAuth.csrfToken);

The operator side read it from the cookie instead. Two patterns for the same job in one codebase, and only one of them survives the move to a second domain.

I had not noticed the inconsistency, because both patterns work on localhost. The divergence was invisible until the environment made one of them wrong. If I had been disciplined about doing the same thing the same way everywhere, the operator side would have inherited the working pattern by accident.

The fix

Three changes.

The backend now returns the token in the response body, not only in the cookie. The service was already generating it, including the rotated one on refresh. The controller was simply dropping it on the floor. The landlord admin endpoints had been doing this correctly all along.

The frontend stores it and sets the header once, centrally, instead of per endpoint:

prepareHeaders: (headers, { getState }) => {
  const { accessToken, csrfToken } = getState().hqleadAuth;
  if (accessToken) headers.set("Authorization", `Bearer ${accessToken}`);
  if (csrfToken) headers.set("x-csrf-token", csrfToken);
  return headers;
}

That detail matters more than it looks. Endpoint-level headers override prepareHeaders, so every per-call getCookie had to be removed. Left in place, they would have overridden a working header with undefined and the bug would have survived its own fix.

The silent refresh carries the rotated token through. Refresh issues a new CSRF token as well as a new access token. Without passing it on, the retried request would send a stale one and fail for a completely different reason.

I shipped it and it still didn't work

The backend went out. I confirmed with curl that login now returned csrfToken. I tried the form again. Same error.

The frontend login handler picks fields off the response by hand:

setCredentials({
  hqLead: response.hqLead,
  accessToken: response.accessToken
})

It was still dropping the field I had just added. The slice was right, the base query was right, the backend was right, and the one line that connected them was not.

I include this because the tidy version of this story — found it, understood it, fixed it — is not what happened. What happened is that I fixed the mechanism and missed the wiring, and only found out by testing against production again.

What the tests did and didn't do

Two backend tests failed when I added the field, because they asserted the exact shape of the login and refresh response bodies. That is correct behaviour. They were pinning a contract, and I had changed it. I updated them with a comment explaining why the token has to be in the body, so that the next person to look does not quietly delete it as redundant.

But notice where the tests were and were not. They covered the backend contract tightly enough to notice a new field. Nothing tested the frontend wiring at all — and that is where the bug lived, both before the fix and after the first attempt at it.

One more thing the screen told me that wasn't true

The same afternoon I armed two-factor authentication on a new account, entered the six-digit code, and watched the form do nothing at all. No error, no confirmation. I assumed it had worked.

It had not. The API said so plainly:

{ "mfa_enabled": true, "mfa_verified": false }

A failed verification renders no message. The enrolment sits half-finished, the account is not protected, and the screen looks exactly like success.

Different bug, same shape as the first one: what the interface shows you and what the system actually holds are two different things, and only one of them is authoritative. I now check the second one.

The pattern underneath

Everything here comes down to one habit. When something looks right, go and ask the system directly rather than believing the screen, the test output, or your own reasoning from an hour ago.

Read the cookie jar. Query the status endpoint. Walk the flow as a customer, in production, clicking the things a customer clicks. It takes twenty minutes and it finds the class of bug that no amount of careful coding prevents, because the bug is not in the code.