Admiral Web QA Console

1. What this is

Admiral Web QA is a shared toolkit that checks websites the same way every time.

• One profile per site - a folder under profiles/ with JSON settings (URL, routes, selectors, budgets, tracking).
• Shared test suites - smoke, accessibility, SEO, tracking, PageSpeed, and more. You do not copy Playwright tests into each app repo.
• This console - a local control panel that runs the same runner as the CLI. Add a site, queue a run, read results, export findings.

2. Start here

1. In a terminal, go to the toolkit folder (adm-web-qa).
2. Run npm run console.
3. Open https://adm-qa-app.development-server.dev/ in your browser.

Or start with Docker: docker compose up console.

Layout: home is the Sites table with an Active tests sidebar. Use the footer links for How to use and About limits. Open a site for its own results, runs, and report page.

3. Sites table

Each row is a website profile. You can open the site, run eligible suites, open the expanded report, or delete the profile (also removes run history, reports, and related artifacts).

1. Click Add website at the top to open the modal.
2. Paste the site URL (no trailing slash needed), optional name, and platform (or auto-detect).
3. Submit - the console creates profiles/<key>/ and discovers routes (public crawl/sitemap, plus login-gated pages when credentials are configured).
4. Click Open on a row (or the site name path /sites/<key>) to enter that website workspace.

Delete permanently removes that profile folder. Confirm carefully.

4. Routes

Routes are the pages the suites will visit. Smoke, a11y, and SEO all depend on this list.

1. Open a site first.
2. Open the Routes tab and review the list.
3. Click Rediscover routes if the site gained new pages or discovery failed earlier.

How discovery works (when you add a website or click Rediscover):

• Local Laravel - matching Laragon folder with artisan / routes/web.php → all static GET routes into routes.json.
• Live sitemap + crawl - same-origin public pages into routes.json (smoke/seo/a11y).
• Authenticated discovery - when login is configured and a session exists, Rediscover crawls while logged in and confirms login-gated pages into routes.auth-discovered.json (any path shape, not only /admin). Used by the auth suite.

Login-walled apps often show only a handful of public routes until authenticated discovery runs. Use Requires login when adding the site, or npm run qa:auth then Rediscover.

Sensitive paths (checkout, cart, payment, register, logout) may land in a needs-approval file. Do not invent checkout or payment tests unless someone approved a safe path.

5. Run your first check

Always start with suite smoke and pack smoke. Fix smoke before running heavier suites.

1. Open the site.
2. Under Run, set Pack = smoke, Suite = smoke.
3. Click Start run.
4. Watch the batch/running panel until it finishes.
5. Stay on the Results tab for that website.

The Recommended next tests panel on each site reflects profile-wide coverage gaps and offers one-click runs for perf, security, checkout, crux, and retest-perf where eligible.

Use Run all tests to queue every eligible suite for this site (one run per suite). Missing security scope, load scenarios, RUM config, and tracking defaults are auto-prepared. New sites queue run-all automatically after add.

6. Suites explained

Pick one suite per run (or use CLI all later). Plain meaning of each:

Axe is not a full WCAG certificate. GTM script presence is not proof an event fired. See About limits for more.

7. Packs

Pack ≠ suite. Pack (smoke / regression / full) is a label for ops — release checklist and run metadata. It does not by itself change which Playwright checks run. You still choose the suite separately.

• smoke - first pass / quick gate labeling.
• regression - after changes when you expect stable checks (label only unless you also pick heavier suites).
• full - broader release-style labeling; Run all tests often queues eligible suites under pack full.

Example: Pack = regression + Suite = smoke still runs only the smoke suite. To get a11y/SEO/tracking, pick those suites (or Run all).

7b. Smoke pass criteria (Playwright steps)

From the smoke storefront test, each route (for example home) is checked roughly like this:

• "… should return an OK response" - page.goto finished with an HTTP 2xx final response. Fails on null response, 4xx, or 5xx. A soft-404 that still returns 200 can pass this step.
• Expect toBeVisible locator('body') - the page <body> exists and is visible (not hidden / zero-size) within the default timeout.
• Also: non-empty document.title, no body horizontal overflow, and no actionable console.error / uncaught page errors (minus allowlist patterns in selectors.json).

Together: the URL loaded successfully and produced a visible HTML page, without obvious JS/layout smoke failures.

7c. Login / credentials

Default smoke does not fill username/password. It opens each route unauthenticated unless a saved session exists.

• When Add website has Requires login checked, the console saves encrypted credentials in profiles/<key>/credentials.enc.json and tries to sign in immediately.
• The master encryption key is QA_CREDENTIALS_KEY. Generate it once and keep it in the runtime secret store; the console will not write keys to disk.
• A successful login writes artifacts/auth/<profile>.json. Playwright auto-loads that storage state for suites. Do not commit auth storage or credential files.
• If a protected URL ends as non-2xx (401/403 or failed redirect) without a session, the OK-response step fails — expected without auth setup.
• Re-login later with npm run qa:auth (uses encrypted credentials, or legacy passwordEnv).
• Rediscover routes with a saved session also runs authenticated discovery: it crawls nav links while logged in, then confirms which paths redirect to login when anonymous. Protected pages are written to routes.auth-discovered.json — any path shape, not only /admin.
• Run the auth suite (npm run qa -- --profile <key> --suite auth) to exercise those login-gated routes. npm run qa:auth only creates the session file; it is not the suite.

7d. Screenshots

• Smoke / most Playwright suites - screenshots, traces, and video are kept only on failure (only-on-failure / retain-on-failure).
• Passed smoke tests - no automatic screenshot.
• Tracking suite - saves final-screenshot.png per scenario as evidence (including on pass).
• Report export may PNG the HTML report — that is not a per-test pass screenshot.

8. Results (per website)

1. Results always belong to the open website (/sites/<key>).
2. Read the tiles: Status, Health, Passed, Failed, Findings.
3. Read the health caveat - health is an ops metric, not sole release truth.
4. Open Playwright report for the interactive HTML report in a full-height console page (/sites/<key>/playwright-report).
5. Open Allure report (when present) for Allure Report 3 Awesome results. Prefer the console link or npm run report:allure:open — do not rely on opening the raw HTML via file://.
6. Click Open expanded report for the full visual audit page (not an iframe).
7. Use the Tests table: filter by passed/failed, paginate, click a row for detail and file preview.
8. Under Findings: triage classifications, select rows, then Export selected to ClickUp when ready.

8b. Expanded QA report page

Opens as a real page at /sites/<key>/reports/<runId> with the console header and a light dashboard (stats, suite scoreboard, charts, findings). Downloads still write under artifacts/reports/<runId>/.

1. Open a site with at least one completed run.
2. Click Open expanded report (or use Open report from the Sites table).
3. Use Generate / refresh to rebuild an expanded (profile-wide) MD/HTML/PNG/PDF package.
4. Download Markdown or the zip package from the toolbar.
5. Use the back link to return to the website workspace.

CLI equivalent: npm run report:full -- --profile your-key --expanded. Numbers come from artifacts/ only — the report never invents findings.

9. History

Past runs for this website only appear as a list. Click one to load that run’s summary into Results.

10. Terminal

This is not a free shell. Only allowlisted project commands run (for example qa, typecheck, adapters).

1. Click a chip for a common command, or type something like qa --profile your-key --suite smoke.
2. Press Run - live stdout/stderr streams in the panel.
3. Use Stop to cancel; Clear to wipe the log view.

Prefer the website Start run / Run all tests queue for structured jobs. Use Terminal when you want a live log stream.

11. ClickUp

1. Token comes from server env CLICKUP_API_TOKEN - never paste tokens into tickets or the UI.
2. Set or load a Default list ID and save defaults.
3. Export only from Results (selected findings). The console does not auto-create tickets.

12. Quality Ops

• Release checklist - layered readiness checks for the profile (not WCAG/SEO/tracking certification).
• Ops KPIs / scorecard - findings, waivers, last-run metrics - not a site health certificate.
• Compare to previous - see what changed vs the prior run.
• Add waiver - suppress a known finding with reason, approver, and expiry.

13. About limits

Open the footer About limits link for evidence boundaries: what each suite does not prove, suite caveats, and budget enforcement mode.

14. Safety rules

• Prefer public pages first.
• Do not submit forms, create accounts, start checkout, or process payment unless a controlled path was explicitly approved.
• Do not invent findings without artifacts. Do not commit secrets, auth storage, or personal data from artifacts.
• For local Laragon *.test HTTPS, native runs often need QA_IGNORE_HTTPS=true (PowerShell: $env:QA_IGNORE_HTTPS="true").
• Keep tracking strict mode off until the taxonomy is baselined.

15. CLI cheat sheet

Same runner outside the UI (from the toolkit root):

npm run qa -- --profile your-key --suite smoke
npm run qa -- --profile your-key --suite a11y
npm run report:full -- --profile your-key
npm run discover:routes -- --url https://example.com --key your-key --write
npm run init:profile -- --key acme --url https://acme.example --platform shopify

Docker (hosted URLs): set QA_PROFILE and BASE_URL to the profile URL when using Compose.

16. If something fails

Classify before fixing randomly:

• Profile defect - wrong URL, missing route, bad selector, incomplete taxonomy. Fix JSON under profiles/, then re-run smoke.
• Product defect - real bug or regression on the site. File a finding with evidence paths.
• Environment - DNS, SSL, site down, wrong BASE_URL override, Docker pointing at another site.

Smoke green (or understood failures) first - then expand to a11y, seo, tracking, pagespeed.