Evidence
Public results of the Veezee tool-selection eval: does a fresh-context model pick the right Veezee tool, emit a valid first call, and read a real response envelope correctly? Machine-readable twin: /evidence/tool-selection-eval.json.
Method#
Anthropic tool-eval rubric, three axes, graded by three fresh-context subagents.
Picker sees ONLY the tool names + descriptions and a realistic user ask, and must name the right tool with no args. First-call-success sees names + descriptions + params and must emit the exact first call (tool + JSON args), validated against the frozen zod schemas. Response-usability sees a real production response envelope and must answer the user's question from it and name its next action.
No agent saw the expected answers. All results reflect the shipped tool descriptions as of the run date.
Models: picker Haiku (the smallest model the descriptions must survive), first-call Sonnet, response-usability Sonnet. Run date: 2026-07-11.
Scope: what this does and doesn't measure#
This eval measures tool-selection and first-call / response-envelope usability only: whether a model picks the right tool for a realistic ask, emits a valid first call, and can read and act on a real response envelope. It does NOT measure data quality, coverage, or freshness, and it does NOT measure API availability or latency. It covers the LinkedIn toolset (five LinkedIn tools plus get_usage); the Reddit and X tools shipped after this run and have not yet been through this eval.
Picker: 13/13#
| Case | Ask | Expected | Selected | Result |
|---|---|---|---|---|
| P1 | Find the LinkedIn profile of the CTO of Datadog. | linkedin_search_people | linkedin_search_people | pass |
| P2 | What has Anthropic been posting on LinkedIn recently? | linkedin_get_posts | linkedin_get_posts | pass |
| P3 | Pull the full work history for linkedin.com/in/satyanadella. | linkedin_get_profile | linkedin_get_profile | pass |
| P4 | How many employees does Stripe have on LinkedIn? | linkedin_get_company | linkedin_get_company | pass |
| P5 | A user pasted this link, what is it? https://www.linkedin.com/company/openai/?utm_source=chatgpt | linkedin_resolve_url or linkedin_get_company | linkedin_resolve_url | pass |
| P7 | How many credits do I have left? | get_usage | get_usage | pass |
| P8 | Find software engineers at Shopify who used to work at Amazon. | linkedin_search_people | linkedin_search_people | pass |
| P9 | Get the education history of williamhgates. | linkedin_get_profile | linkedin_get_profile | pass |
| P10 | I only have the website vercel.com. Get their LinkedIn company info. | linkedin_get_company | linkedin_get_company | pass |
| P11 | Did Bill Gates post anything about malaria this month? | linkedin_get_posts | linkedin_get_posts | pass |
| P12 | Look up the profile of the person named 'Jordan Rivera' who works at Datadog. | linkedin_search_people | linkedin_search_people | pass |
| P13 | Check whether my key is still valid and what plan I'm on. | get_usage | get_usage | pass |
| P14 | Read this specific post for me: https://www.linkedin.com/feed/update/urn:li:activity:7123456789/ | linkedin_resolve_url or none | none | pass |
First-call success: 8/8#
| Case | Ask | Tool | Args | Result |
|---|---|---|---|---|
| F1 | Pull the work history for linkedin.com/in/satyanadella. | linkedin_get_profile | {"identifier":"linkedin.com/in/satyanadella","sections":["experience"]} | pass |
| F2 | Find software engineers at Shopify who used to work at Amazon. | linkedin_search_people | {"title":"Software Engineer","current_company":"Shopify","past_company":"Amazon"} | pass |
| F3 | I only have the website vercel.com. Get their LinkedIn company info. | linkedin_get_company | {"identifier":"vercel.com"} | pass |
| F4 | What has Anthropic been posting on LinkedIn recently? | linkedin_get_posts | {"identifier":"anthropic"} | pass |
| F5 | Get the education history of williamhgates. | linkedin_get_profile | {"identifier":"williamhgates","sections":["education"]} | pass |
| F6 | Search up to 30 results for CTOs at Anthropic, but do not spend more than 15 credits on the call. | linkedin_search_people | {"keywords":"CTO Anthropic","title":"CTO","limit":30,"max_credits":15} | pass |
| F7 | Get LinkedIn company info for microsoft.com with a hard 6-credit ceiling. | linkedin_get_company | {"identifier":"microsoft.com","max_credits":6} | pass |
| F8 | (a LinkedIn company URL passed as the identifier) | linkedin_get_company | {} | pass |
Notes on the hard cases: F3 Website domains are first-class identifiers: the domain reaches the company in one call, zero extra turns.
F6 The hard case: a result cap plus a hard credit ceiling. The model followed the description's budget rule (quote 12 <= max_credits 15) and avoided the company-name-filter-plus-3-pages combination the planner rejects; the server's teaching error remains the safety net for agents that miss it.
F7 Correctly predicted from the published prices that domain resolution quotes 8 credits, exceeding max_credits=6, and expected the call to reject uncharged.
F8 Passed the LinkedIn URL straight through as identifier, without misrouting it through linkedin_resolve_url first.
Response usability: 3/3#
| Case | Ask | Envelope source | Result |
|---|---|---|---|
| U1 | How many employees does Microsoft have on LinkedIn, and what industry is it in? | production linkedin_get_company identifier=microsoft | pass |
| U2 | How many credits do I have left and what did my last few calls cost? | production get_usage | pass |
| U3 | Find CTOs at Anthropic; I will want their profiles next. | production linkedin_search_people keywords=CTO current_company=1035 limit=10 | pass |
Note on U3: the eval's own input envelope was a Microsoft-id search mislabeled "Anthropic". The pass is the model catching that mismatch from the envelope's echoed current_company field and refusing to spend credits on wrong-company fetches; the full grading note is in the JSON twin.
30-day availability and latency#
Pending. Synthetics-derived 30-day availability and p50/p95 latency have not accumulated enough data yet. This section will be filled in once that monitoring window closes; no numbers are published here in the meantime.
Verdict#
Descriptions pass the picker axis on a small fast model (Haiku) and the first-call axis on a mid-size model (Sonnet). The hardest cases (a result cap combined with a hard credit ceiling; a website domain as the only identifier) pass on the first call with the shipped descriptions.