Rubric rubric/v1
100 points, seven items. Every point maps to something the probe observed; items it could not observe score 0 and are listed as such instead of being guessed. Two items use a cheap LLM judge (Sonnet-class, capped at two calls per scan and $0.05 per host per day); the rest is deterministic. Grades: A ≥ 85, B ≥ 70, C ≥ 55, D ≥ 40, F below.
| Item | Max | How points are awarded |
|---|---|---|
| Signup form an agent can fill | 20 | 20 email + password form; 12 email-only (magic link); 8 OAuth buttons only; 0 nothing found on the URL or its linked signup/register pages. |
| No CAPTCHA / bot wall | 15 | 15 when no reCAPTCHA / hCaptcha / Turnstile / Arkose script or widget guards the form; 0 when one does. The probe never solves CAPTCHAs — it records the vendor and stops. |
| Signup completes without a human | 20 | The probe submits the form with a clearly labelled probe address (agentwares-probe-…@example.com). 20 landed on an authenticated page; 10 asked for email verification; 0 blocked or validation error; not observed when no form or a CAPTCHA. |
| Terms acceptable in the flow | 10 | 10 explicit checkbox; 8 'by signing up you agree' text; 4 terms only linked elsewhere; 0 none. |
| Terms allow automated accounts(judged) | 5 | 5 when the terms page does not forbid automated / agent-operated accounts (or allows them via API); 0 when it does. The only judgment call in the rubric. |
| API key minting | 20 | 20 a create/generate-key control was reachable after signup; 12 a keys page exists but the control needs a gate (billing, 2FA); 6 keys are only mentioned in docs; 0 nothing. |
| Docs an agent can act on(judged) | 10 | 4 /llms.txt returns 200; 3 /openapi.json returns 200; 3 the docs explain how a program obtains credentials and sends them (judged). |
Guardrails
- Only domains that proved ownership (DNS TXT, meta tag or .well-known token) are probed.
- 3 scans per host per day; 10 per client per hour; 30 per hour overall.
- The probe identifies itself (User-Agent
AgentwaresReadinessProbe), uses@example.comaddresses, and never solves CAPTCHAs. - Results, evidence and the fix list are public at
/r/<id>; raw HTML is not stored, only extracted text (≤ 8 KB per page).