Rubric rubric/v1
100 points, seven items. Every point maps to something the probe observed; items it could not observe score 0 and are listed as such instead of being guessed. Two items use a cheap LLM judge (Sonnet-class, capped at two calls per scan and $0.05 per host per day); the rest is deterministic. Grades: A ≥ 85, B ≥ 70, C ≥ 55, D ≥ 40, F below.
| Item | Max | How points are awarded |
|---|---|---|
| Signup form an agent can fill | 20 | 20 email + password form; 12 email-only (magic link); 8 OAuth buttons only; 0 nothing found on the URL or its linked signup/register pages. |
| No CAPTCHA / bot wall | 15 | 15 when no reCAPTCHA / hCaptcha / Turnstile / Arkose script or widget guards the form; 0 when one does. The probe never solves CAPTCHAs — it records the vendor and stops. |
| Signup completes without a human | 20 | The probe submits the form with a clearly labelled probe address (agentwares-probe-…@example.com). 20 landed on an authenticated page; 10 asked for email verification; 0 blocked or validation error; not observed when no form or a CAPTCHA. |
| Terms acceptable in the flow | 10 | 10 explicit checkbox; 8 'by signing up you agree' text; 4 terms only linked elsewhere; 0 none. |
| Terms allow automated accounts(judged) | 5 | 5 when the terms page does not forbid automated / agent-operated accounts (or allows them via API); 0 when it does. The only judgment call in the rubric. |
| API key minting | 20 | 20 a create/generate-key control was reachable after signup; 12 a keys page exists but the control needs a gate (billing, 2FA); 6 keys are only mentioned in docs; 0 nothing. |
| Docs an agent can act on(judged) | 10 | 4 /llms.txt returns 200; 3 /openapi.json returns 200; 3 the docs explain how a program obtains credentials and sends them (judged). |
Guardrails
- Only domains that proved ownership (DNS TXT, meta tag or .well-known token) are probed.
- 3 scans per host per day; 10 per client per hour; 30 per hour overall.
- The probe identifies itself (User-Agent
AgentwaresReadinessProbe), uses@example.comaddresses, and never solves CAPTCHAs. - Results, evidence and the fix list are public at
/r/<id>; raw HTML is not stored, only extracted text (≤ 8 KB per page).
Rubric changes
Changed on 2026-09-07. The probe used to attempt a signup only when the form had both an email and a password field, and scored everything else zero on “completes without a human” without trying. That is a false negative for every passwordless or optional-email signup — the shape an agent-first signup usually takes. It now attempts any form that is not behind a CAPTCHA and scores the outcome: a mailbox round-trip is still recorded as email_verification, and only a real completion counts as success. A credential returned on the page after submitting counts for key minting, because that is the question the item asks.
In fairness we should say: this change raised our own score, because agentcheck has exactly that kind of signup. The rubric is versioned and applies to every scan; older results keep the rubric they were scored under.