Live vendor tests

The opt-in lane that calls Stripe and the AI providers for real - what it proves, what it costs, and how it gates.

Every other suite answers to a fake — deterministic, and blind: a fake never renames a field, retires a model, or starts refusing your account. This lane is the only one that would notice.

The two lanes

	Fast lane	Live lane
Commands	`pnpm test`, `pnpm e2e:ci`	`pnpm test:live`, `pnpm e2e:live`
Runs	every commit	before a release, and on `main` in CI
Stripe	loopback fake	your test-mode account
AI providers	local fake	the real API
Needs keys	no	yes - reports what it skipped without them

What the live lane proves

Suite	Catches
`packages/payments/tests-live`	a checkout parameter Stripe now rejects; a retried auto top-up charging twice; an upgrade's proration no longer invoiced immediately; a signature no longer verifying
`packages/api/tests-live`	your default model leaving the catalog; the exact-cost field moving, which silently bills every turn from a token estimate and leaves you absorbing the fees
`apps/web/e2e/*.live.spec.ts`	a real turn that streams a reply and settles nothing — or far too much — against the account that ran it

Keys

Each is optional. An absent key removes its cases and prints one not run: line; a key that is present and refused fails.

Variable	Enables
`STRIPE_SECRET_KEY`	the Stripe suite - `sk_test_` only
`STRIPE_WEBHOOK_SECRET`	webhook signature verification
`OPENROUTER_API_KEY`	the built-in model turn, and the live chat spec
`FIRECRAWL_API_KEY` / `PARALLEL_API_KEY` / `TINYFISH_API_KEY`	the web search and extract tools

A live-mode Stripe key aborts the run. The suite attaches cards and confirms off-session charges; against an sk_live_ key those are real charges against real people. It checks the prefix before any API call and fails loudly, naming the variable and the mode it found. To opt out, leave STRIPE_SECRET_KEY unset — that skips the lane quietly.

What a run costs

Stripe test mode moves no money (about 25 API calls). The AI providers charge: ~$0.014 Firecrawl, ~$0.006 Parallel, nothing at all for TinyFish, and fractions of a cent for the capped model turn. A new case here is a recurring bill, not a one-off.

What it leaves in your Stripe account

Everything it creates is tagged generatesaas-live-suite. A run that fails during setup unwinds what it already made.

Removed: the customer and its card, the subscription, open checkout sessions, unconfirmed PaymentIntents.
Archived, because Stripe has no delete: one product and three prices.
Permanent: confirmed PaymentIntents, and the invoices the subscription raised.

Your test account therefore accumulates, unavoidably rather than as a leak: Stripe cannot delete a price that has ever been used, so every run leaves one more archived product, three more archived prices, and confirmed PaymentIntents that stay forever. Archived objects are hidden from the dashboard's default views and can never be charged against. Nothing accumulates in live mode — the suite refuses to run there.

Running it

export STRIPE_SECRET_KEY=sk_test_...   # and whichever others you hold
pnpm test:live                          # vendors + Stripe; no database, no services
pnpm infra && pnpm e2e:live             # the browser half, against a real server

Making it a gate

An opt-in suite exits 0 whether it smoked every vendor or found none, so the exit code alone proves nothing — a revoked key looks like a project that never had one. Set the flag wherever credentials are supposed to exist:

GENERATESAAS_LIVE_REQUIRED=1 pnpm test:live

With it, a run that executes nothing fails and names what was missing; without it, that run reports and passes. Each suite also writes test-results/live-status.json:

Field	Means
`ran`	how many cases actually executed
`required`	the machine was equipped - credentials present, provider configured
`notRun`	one line per case that could not run, and why

required: true beside ran: 0 is a broken gate, not an idle one.