# Evidence - Veezee

Public results of the B5 tool-selection eval: does a fresh-context model pick the right Veezee tool, emit a valid first call, and read a real response envelope correctly? Machine-readable twin: [/evidence/tool-selection-eval.json](/evidence/tool-selection-eval.json).

## Method

Anthropic tool-eval rubric, three axes, graded by three fresh-context subagents. Picker sees ONLY the seven tool names + descriptions and a realistic user ask, and must name the right tool with no args. First-call-success sees names + descriptions + params and must emit the exact first call (tool + JSON args), validated against the frozen zod schemas. Response-usability sees a real production response envelope and must answer the user's question from it and name its next action. No agent saw the expected answers.

Models: picker Haiku (the floor host), first-call Sonnet, response-usability Sonnet. Run date: 2026-07-11.

## Scope: what this does and doesn't measure

This eval measures tool-selection and first-call / response-envelope usability only: whether a model picks the right tool for a realistic ask, emits a valid first call, and can read and act on a real response envelope. It does NOT measure LinkedIn data quality, coverage, or freshness, and it does NOT measure API availability or latency.

## Picker: 14/14

| Case | Ask | Expected | Selected | Result |
| --- | --- | --- | --- | --- |
| P1 | Find the LinkedIn profile of the CTO of Datadog. | search_people | search_people | pass |
| P2 | What has Anthropic been posting on LinkedIn recently? | get_posts | get_posts | pass |
| P3 | Pull the full work history for linkedin.com/in/satyanadella. | get_profile | get_profile | pass |
| P4 | How many employees does Stripe have on LinkedIn? | get_company | get_company | pass |
| P5 | A user pasted this link, what is it? https://www.linkedin.com/company/openai/?utm_source=chatgpt | resolve_url or get_company | resolve_url | pass |
| P6 | I don't have a Veezee API key yet. Get me set up. | provision | provision | pass |
| P7 | How many credits do I have left? | get_usage | get_usage | pass |
| P8 | Find software engineers at Shopify who used to work at Amazon. | search_people | search_people | pass |
| P9 | Get the education history of williamhgates. | get_profile | get_profile | pass |
| P10 | I only have the website vercel.com. Get their LinkedIn company info. | get_company | get_company | pass |
| P11 | Did Bill Gates post anything about malaria this month? | get_posts | get_posts | pass |
| P12 | Look up the profile of the person named 'Jordan Rivera' who works at Datadog. | search_people | search_people | pass |
| P13 | Check whether my key is still valid and what plan I'm on. | get_usage | get_usage | pass |
| P14 | Read this specific post for me: https://www.linkedin.com/feed/update/urn:li:activity:7123456789/ | resolve_url or none | none | pass |

## First-call success: 7/7 valid, 6/7 optimal on the first try

| Case | Ask | Tool | Args | Optimal first try |
| --- | --- | --- | --- | --- |
| F1 | Pull the work history for linkedin.com/in/satyanadella. | get_profile | `{"identifier":"linkedin.com/in/satyanadella","sections":["experience"]}` | yes |
| F2 | Find software engineers at Shopify who used to work at Amazon. | search_people | `{"title":"Software Engineer","current_company":"Shopify","past_company":"Amazon"}` | yes |
| F3 | I only have the website vercel.com. Get their LinkedIn company info. | get_company | `{"identifier":"vercel"}` | yes |
| F4 | What has Anthropic been posting on LinkedIn recently? | get_posts | `{"identifier":"anthropic"}` | yes |
| F5 | Get the education history of williamhgates. | get_profile | `{"identifier":"williamhgates","sections":["education"]}` | yes |
| F6 | Search up to 30 results for CTOs at Anthropic, but do not spend more than 15 credits on the call. | search_people | `{}` | no |
| F7-A5 | HISTORICAL (pre-WAVE3): get_company identifier='microsoft.com' returns a teaching INVALID_INPUT; the agent must produce the corrected call in one turn. | get_company | `{"identifier":"microsoft"}` | yes |

**F6 needed a description fix.** The one systematic weakness this eval found. v1 and v2 tool descriptions let the model combine a company NAME filter with limit=30, which the planner rejects (name resolution plus 3 pages exceeds the internal fetch budget): recoverable via the server error, but a first-call failure. Fixed in v3 with an imperative description rule; the server error remains the safety net for agents that ignore it.

| Version | Args | Outcome |
| --- | --- | --- |
| v1 (no caveat) | `{"title":"CTO","current_company":"Anthropic","limit":30,"max_credits":15}` | rejected by the planner (name filter + limit=30 combination) |
| v2 (polite caveat added to description) | `{"title":"CTO","current_company":"Anthropic","limit":30,"max_credits":15}` | unchanged: same rejected combination |
| v3 (imperative caveat: Do NOT combine a company NAME filter with limit=30...) | `{"keywords":"CTO Anthropic","title":"CTO","limit":30,"max_credits":15}` | valid: avoided the rejected combination, quote 12 <= max_credits 15; shipped as the live description |

**F7-A5 is historical.** Historical; superseded 2026-07-11 by the WAVE3 F3 domain-emulation change, under which 'microsoft.com' now resolves directly with zero extra turns (see the rerun below).

## Response usability: 3/3

| Case | Ask | Envelope source | Result |
| --- | --- | --- | --- |
| U1 | How many employees does Microsoft have on LinkedIn, and what industry is it in? | production get_company identifier=microsoft | pass |
| U2 | How many credits do I have left and what did my last few calls cost? | production get_usage | pass |
| U3 | Find CTOs at Anthropic; I will want their profiles next. | production search_people keywords=CTO current_company=1035 limit=10 | pass |

## WAVE3 domain-emulation rerun (2026-07-11)

After the get_company description change that made website domains first-class identifiers (WAVE3 F3), the affected picker and first-call cases were rerun.

Picker: 3/3

| Ask | Selected | Result |
| --- | --- | --- |
| bare domain (vercel.com) | get_company | pass |
| employee-count-behind-domain (stripe.com) | get_company | pass |
| subdomain paste (news.microsoft.com) | get_company | pass |

First-call: 3/3

| Case | Ask | Tool | Args | Result |
| --- | --- | --- | --- | --- |
| F3-rerun | I only have the website vercel.com. Get their LinkedIn company info. | get_company | `{"identifier":"vercel.com"}` | pass |
| F3b | Get LinkedIn company info for microsoft.com with a hard 6-credit ceiling. | get_company | `{"identifier":"microsoft.com","max_credits":6}` | pass |
| F3c | (a LinkedIn URL passed as the company identifier) | get_company | `{}` | pass |

Raw per-case model transcripts for this rerun are not separately archived in evidence/tool-evals/transcripts.md (only the F6 description-iteration reruns are); this section reproduces the verdicts recorded in evidence/tool-evals/results.md.

## 30-day availability and latency

Pending. Synthetics-derived 30-day availability and p50/p95 latency have not accumulated enough data yet. This section will be filled in once that monitoring window closes; no numbers are published here in the meantime.

## Verdict

Descriptions pass picker and first-call at the Haiku floor and Sonnet mid-tier. The one systematic weakness found (a company NAME filter combined with limit=30 on search_people) now has an imperative description rule plus a teaching server error. The WAVE3 F3 domain-emulation rerun is green.