Evidence
Public results of the B5 tool-selection eval: does a fresh-context model pick the right Veezee tool, emit a valid first call, and read a real response envelope correctly? Machine-readable twin: /evidence/tool-selection-eval.json.
Method
Anthropic tool-eval rubric, three axes, graded by three fresh-context subagents. Picker sees ONLY the seven tool names + descriptions and a realistic user ask, and must name the right tool with no args. First-call-success sees names + descriptions + params and must emit the exact first call (tool + JSON args), validated against the frozen zod schemas. Response-usability sees a real production response envelope and must answer the user's question from it and name its next action. No agent saw the expected answers.
Models: picker Haiku (the floor host), first-call Sonnet, response-usability Sonnet. Run date: 2026-07-11.
Scope: what this does and doesn't measure
This eval measures tool-selection and first-call / response-envelope usability only: whether a model picks the right tool for a realistic ask, emits a valid first call, and can read and act on a real response envelope. It does NOT measure LinkedIn data quality, coverage, or freshness, and it does NOT measure API availability or latency.
Picker: 14/14
| Case | Ask | Expected | Selected | Result |
|---|---|---|---|---|
| P1 | Find the LinkedIn profile of the CTO of Datadog. | search_people | search_people | pass |
| P2 | What has Anthropic been posting on LinkedIn recently? | get_posts | get_posts | pass |
| P3 | Pull the full work history for linkedin.com/in/satyanadella. | get_profile | get_profile | pass |
| P4 | How many employees does Stripe have on LinkedIn? | get_company | get_company | pass |
| P5 | A user pasted this link, what is it? https://www.linkedin.com/company/openai/?utm_source=chatgpt | resolve_url or get_company | resolve_url | pass |
| P6 | I don't have a Veezee API key yet. Get me set up. | provision | provision | pass |
| P7 | How many credits do I have left? | get_usage | get_usage | pass |
| P8 | Find software engineers at Shopify who used to work at Amazon. | search_people | search_people | pass |
| P9 | Get the education history of williamhgates. | get_profile | get_profile | pass |
| P10 | I only have the website vercel.com. Get their LinkedIn company info. | get_company | get_company | pass |
| P11 | Did Bill Gates post anything about malaria this month? | get_posts | get_posts | pass |
| P12 | Look up the profile of the person named 'Jordan Rivera' who works at Datadog. | search_people | search_people | pass |
| P13 | Check whether my key is still valid and what plan I'm on. | get_usage | get_usage | pass |
| P14 | Read this specific post for me: https://www.linkedin.com/feed/update/urn:li:activity:7123456789/ | resolve_url or none | none | pass |
First-call success: 7/7 valid, 6/7 optimal on the first try
| Case | Ask | Tool | Args | Optimal first try |
|---|---|---|---|---|
| F1 | Pull the work history for linkedin.com/in/satyanadella. | get_profile | {"identifier":"linkedin.com/in/satyanadella","sections":["experience"]} | yes |
| F2 | Find software engineers at Shopify who used to work at Amazon. | search_people | {"title":"Software Engineer","current_company":"Shopify","past_company":"Amazon"} | yes |
| F3 | I only have the website vercel.com. Get their LinkedIn company info. | get_company | {"identifier":"vercel"} | yes |
| F4 | What has Anthropic been posting on LinkedIn recently? | get_posts | {"identifier":"anthropic"} | yes |
| F5 | Get the education history of williamhgates. | get_profile | {"identifier":"williamhgates","sections":["education"]} | yes |
| F6 | Search up to 30 results for CTOs at Anthropic, but do not spend more than 15 credits on the call. | search_people | {} | no |
| F7-A5 | HISTORICAL (pre-WAVE3): get_company identifier='microsoft.com' returns a teaching INVALID_INPUT; the agent must produce the corrected call in one turn. | get_company | {"identifier":"microsoft"} | yes |
F6 needed a description fix. The one systematic weakness this eval found. v1 and v2 tool descriptions let the model combine a company NAME filter with limit=30, which the planner rejects (name resolution plus 3 pages exceeds the internal fetch budget): recoverable via the server error, but a first-call failure. Fixed in v3 with an imperative description rule; the server error remains the safety net for agents that ignore it.
| Version | Args | Outcome |
|---|---|---|
| v1 (no caveat) | {"title":"CTO","current_company":"Anthropic","limit":30,"max_credits":15} | rejected by the planner (name filter + limit=30 combination) |
| v2 (polite caveat added to description) | {"title":"CTO","current_company":"Anthropic","limit":30,"max_credits":15} | unchanged: same rejected combination |
| v3 (imperative caveat: Do NOT combine a company NAME filter with limit=30...) | {"keywords":"CTO Anthropic","title":"CTO","limit":30,"max_credits":15} | valid: avoided the rejected combination, quote 12 <= max_credits 15; shipped as the live description |
F7-A5 is historical. Historical; superseded 2026-07-11 by the WAVE3 F3 domain-emulation change, under which 'microsoft.com' now resolves directly with zero extra turns (see the rerun below).
Response usability: 3/3
| Case | Ask | Envelope source | Result |
|---|---|---|---|
| U1 | How many employees does Microsoft have on LinkedIn, and what industry is it in? | production get_company identifier=microsoft | pass |
| U2 | How many credits do I have left and what did my last few calls cost? | production get_usage | pass |
| U3 | Find CTOs at Anthropic; I will want their profiles next. | production search_people keywords=CTO current_company=1035 limit=10 | pass |
WAVE3 domain-emulation rerun (2026-07-11)
After the get_company description change that made website domains first-class identifiers (WAVE3 F3), the affected picker and first-call cases were rerun.
Picker: 3/3
| Ask | Selected | Result |
|---|---|---|
| bare domain (vercel.com) | get_company | pass |
| employee-count-behind-domain (stripe.com) | get_company | pass |
| subdomain paste (news.microsoft.com) | get_company | pass |
First-call: 3/3
| Case | Ask | Tool | Args | Result |
|---|---|---|---|---|
| F3-rerun | I only have the website vercel.com. Get their LinkedIn company info. | get_company | {"identifier":"vercel.com"} | pass |
| F3b | Get LinkedIn company info for microsoft.com with a hard 6-credit ceiling. | get_company | {"identifier":"microsoft.com","max_credits":6} | pass |
| F3c | (a LinkedIn URL passed as the company identifier) | get_company | {} | pass |
Raw per-case model transcripts for this rerun are not separately archived in evidence/tool-evals/transcripts.md (only the F6 description-iteration reruns are); this section reproduces the verdicts recorded in evidence/tool-evals/results.md.
30-day availability and latency
Pending. Synthetics-derived 30-day availability and p50/p95 latency have not accumulated enough data yet. This section will be filled in once that monitoring window closes; no numbers are published here in the meantime.
Verdict
Descriptions pass picker and first-call at the Haiku floor and Sonnet mid-tier. The one systematic weakness found (a company NAME filter combined with limit=30 on search_people) now has an imperative description rule plus a teaching server error. The WAVE3 F3 domain-emulation rerun is green.