Test sign-up emails and OTP codes in Playwright

End-to-end tests of sign-up flows break on the email step. Teams stub the mailer (so the real email is never tested), share one test mailbox (so parallel tests read each other’s codes), or poll IMAP with a regex (so the suite gets slow and flaky).

A temporary Agentboxd inbox per test avoids all three. It is a real address, so your app sends real mail through its real provider. It belongs to one test, so parallel workers never collide. And waitForVerification returns the code as soon as the message arrives, already extracted. Written against Playwright Test 1.63.

#Install

npm install --save-dev @playwright/test agentboxd
npx playwright install chromium

export AGENTBOXD_API_KEY=mr_...   # a CI secret in your pipeline

Use a separate API key for tests, with the Manage inboxes permissions plus messages:read, or simply Full access in a test-only workspace. Temporary inboxes don’t count toward your plan’s inbox limit; the mail they receive counts toward emails.

#Add an inbox fixture

A Playwright fixture creates the inbox before the test and deletes it after, even when the test fails. Deleting a temporary inbox wipes its messages at once; if a run crashes, the inbox wipes itself at expires_at anyway.

/**
 * A Playwright fixture that gives each test its own temporary Agentboxd inbox and deletes it
 * (with its mail) when the test ends. Temporary inboxes are receive-only and live on their own
 * domain, so test sign-ups never touch your permanent agent addresses.
 */
import { test as base } from '@playwright/test';
import { Agentboxd, type Inbox } from 'agentboxd';

export const mr = new Agentboxd();

export const test = base.extend<{ inbox: Inbox }>({
  // eslint-disable-next-line no-empty-pattern -- Playwright fixtures take a destructured first argument.
  inbox: async ({}, use) => {
    const inbox = await mr.inboxes.createTemporary({ ttlSeconds: 900 });
    await use(inbox);
    await mr.inboxes.delete(inbox.id); // wipe now instead of at expires_at
  },
});

export { expect } from '@playwright/test';

/** Waits for the sign-up email and returns its code; fails the test with a clear message on timeout. */
export async function verificationCode(inbox: Inbox, since: string, opts: { from?: string; timeout?: number } = {}): Promise<string> {
  const v = await mr.messages.waitForVerification(inbox.id, { since, from: opts.from, timeout: opts.timeout ?? 30 });
  if (!v?.code) throw new Error(`no verification code for ${inbox.address} within ${opts.timeout ?? 30} s`);
  return v.code;
}

Temporary addresses live on tmp.agentboxd.com and look like k3v9q2m8x1ab@tmp.agentboxd.com. If your app blocks disposable-email domains, allow that domain in your test environment, or use a permanent inbox with a client_id per test instead (mr.inboxes.create({ client_id: testInfo.testId })).

#Write the test

The one rule: take the timestamp before the click that sends the email, and pass it as since. The wait call only counts mail that arrives after since, and a fast mailer can deliver before your next line runs.

import { expect, test, verificationCode } from './fixtures.js';

test('a new user signs up with an emailed code', async ({ page, inbox }) => {
  await page.goto('/signup');
  await page.getByLabel('Email').fill(inbox.address);

  const since = new Date().toISOString(); // before the click that sends the email
  await page.getByRole('button', { name: 'Sign up' }).click();
  await expect(page.getByRole('heading', { name: 'Check your email' })).toBeVisible();

  const code = await verificationCode(inbox, since, { from: 'acme.example' });
  expect(code).toMatch(/^\d{6}$/);

  await page.getByLabel('Verification code').fill(code);
  await page.getByRole('button', { name: 'Verify' }).click();
  await expect(page.getByRole('heading', { name: 'Welcome to Acme' })).toBeVisible();
});

The from filter is a case-insensitive substring of the sender, so a stray email can’t satisfy the wait. For magic links, read v.link instead of v.code and page.goto(v.link). Each result also has a confidence from 0 to 1: codes found by pattern matching are capped at 0.7, and our classifier raises that to at least 0.95 when it confirms the message is a verification email.

#Configure and run it

import { defineConfig, devices } from '@playwright/test';

// APP_URL points at the app under test. Without it, the demo app in this folder is started.
const appUrl = process.env.APP_URL;

export default defineConfig({
  testDir: '.',
  // Waiting for an email takes a few seconds; leave room for it on top of the page steps.
  timeout: 60_000,
  retries: process.env.CI ? 1 : 0,
  use: { baseURL: appUrl ?? 'http://localhost:4400', trace: 'retain-on-failure' },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
  ...(appUrl
    ? {}
    : {
        webServer: {
          command: 'npx tsx playwright-email-verification/demo-app.ts',
          url: 'http://localhost:4400/health',
          reuseExistingServer: true,
          cwd: '..',
        },
      }),
});

APP_URL=https://staging.example.com npx playwright test

Give the test a longer timeout than a pure UI test: most transactional mail arrives within a few seconds, but providers occasionally queue. The helper waits up to 30 seconds per call, and one call can wait at most 60.

#Running it in CI

• Store AGENTBOXD_API_KEY as a secret and pass it to the test step.
• Parallel workers are fine: every test has its own inbox. A workspace can have 25 active temporary inboxes and create 60 an hour, so size workers and retries with that in mind, or use permanent inboxes with a client_id per test for very large suites.
• Keep traces on failure (trace: "retain-on-failure"); the email step is then visible next to the page steps.
• Don’t assert on the email’s exact wording in the same test. If you want to check the template, fetch the message with mr.messages.get(v.message_id) and assert on subject and extracted_text in a separate test.

#Why not a fake SMTP server?

A local catch-all such as a development SMTP sink is great for unit-level checks of your templates, and you should keep it. It can’t tell you whether the real provider sends the mail, whether the link in production points to the right host, or whether the email survives a real spam check. The temporary inbox receives exactly what a user would, over the internet, with SPF, DKIM and DMARC results attached to the message.

The runnable copy of this guide (examples/guides/playwright-email-verification/ in the Agentboxd repository) includes a tiny sign-up app, so you can run the test before pointing it at your own. The same flow for AI agents instead of tests is in Agents that sign up for services, and the QA side in QA and test automation.