Ask the handbookDemo with a fictional company

25/25

questions passed every check on the latest run.

Run 17 October 2026 at 03:05 UTCModel meta/llama-3.3-70b-instruct-fp8-fastAgainst demo-doc-chat.horatiu-330.workers.dev

Why a chatbot needs a test set

A demo that answers three questions well proves little. What a team needs to know before shipping is how often it finds the right page, whether its citations hold up, and whether it stays quiet when the documents do not cover a question. Those are measurable, so this demo measures them on every change.

The set has 20 questions the handbook answers, each tagged with the document that holds the answer and the fact the answer must state, and 5 questions it must refuse: near misses such as a company car (the handbook only mentions company cards), facts it never states, and off-topic requests.

What each check means

Retrieved
The expected document is among the 4 passages handed to the model. If retrieval misses, the answer cannot be right.
Cited
The answer cites at least one passage from the expected document, and no citation points at a passage that was not retrieved.
Fact
The answer contains the expected fact, such as “25 working days” or “premium economy”. Checked by string match, not by another model.
Refusal
Questions outside the handbook get “not in the handbook”; questions inside it get an answer.

How a run works

npm run eval sends every question to the deployed API, the same endpoint the chat uses, and scores the replies with plain code. The results are committed as JSON, and this page is built from that file. Runs are kept rare on purpose: each one spends part of the free daily AI allowance.

npm run eval            # every question, deployed API
npm run eval:retrieval  # retrieval only, no model calls

How refusals work

Two gates. If even the closest passage scores below 0.5 cosine similarity, the question is refused without calling the model at all. Otherwise the model is told to reply with a fixed token when the passages do not answer; the server catches that token before it is shown and turns it into a refusal.

Run history

  1. 17 October 2026 at 03:05 UTC25/25 passedretrieval 20/20 · citations 20/20 · facts 20/20 · refusals 25/25

    After widening those two expected facts. Same prompt, models and handbook; the answers to the two questions were again correct and now count as such.

  2. 17 October 2026 at 02:55 UTC23/25 passedretrieval 20/20 · citations 20/20 · facts 18/20 · refusals 25/25

    First run. Both failures were correct answers that the string check missed: “a written description is enough” for a €15 taxi, and “business class is not available”. The expected facts were too narrow for the questions, so they were widened to accept those wordings before the next run.

Every question, every check

Open a row to read the answer the model gave on this run.

QuestionRetrievedCitedFactRefusalOverall
How many days of annual leave do I get?leave-days · expects leave-and-time-off
Answer

You get 25 working days of paid annual leave per leave year [1]. If you are part-time, you receive this entitlement pro rata to your contracted hours [1].

Cited: leave-and-time-off

RetrievedPassCitedPassFactPassRefusalPassOverallPass
Can I carry unused holiday into next year?leave-carry-over · expects leave-and-time-off
Answer

You can carry up to 5 unused days into the next leave year [1]. Carried-over days must be used by 31 March, after which they expire [1].

Cited: leave-and-time-off

RetrievedPassCitedPassFactPassRefusalPassOverallPass
When do I need a doctor's note if I'm off sick?sick-note · expects leave-and-time-off
Answer

You need a doctor's note from the fourth day of absence [1].

Cited: leave-and-time-off

RetrievedPassCitedPassFactPassRefusalPassOverallPass
How long is paid parental leave for the primary caregiver?parental-primary · expects leave-and-time-off
Answer

The primary caregiver of a new child receives 20 weeks of parental leave at full pay [1].

Cited: leave-and-time-off

RetrievedPassCitedPassFactPassRefusalPassOverallPass
What's the deadline for submitting an expense claim?expense-deadline · expects expenses
Answer

You need to submit every claim within 30 days of the date on the receipt [1]. Claims older than 30 days need written approval from your department lead [1].

Cited: expenses

RetrievedPassCitedPassFactPassRefusalPassOverallPass
Do I need a receipt for a 15 euro taxi ride?receipt-threshold · expects expenses
Answer

No, for smaller amounts such as a short taxi ride, a written description is enough [1].

Cited: expenses

RetrievedPassCitedPassFactPassRefusalPassOverallPass
How much can I spend on meals per day when I travel for work?meal-cap · expects expenses or business-travel
Answer

You can spend up to €45 per day on meals when you travel for work [1][2]. This is a cap, not an allowance, so you should claim what you actually spent [1].

Cited: expenses, business-travel

RetrievedPassCitedPassFactPassRefusalPassOverallPass
What mileage rate is paid when I use my own car for work?mileage · expects expenses
Answer

You are paid €0.30 per kilometre when you use your own car for a business journey [1].

Cited: expenses

RetrievedPassCitedPassFactPassRefusalPassOverallPass
Which days am I expected to be in the office?anchor-days · expects remote-and-hybrid-work
Answer

You are expected to be in the office on Tuesday and Wednesday, which are designated as anchor days [1]. Teams can agree to swap one anchor day for another weekday, but this must be a team-wide change and documented in the team handbook page [1]. Your first day also includes a welcome session and lunch with your team in the office [4].

Cited: remote-and-hybrid-work, onboarding

RetrievedPassCitedPassFactPassRefusalPassOverallPass
How long can I work from another country?work-abroad · expects remote-and-hybrid-work
Answer

You can work from a country other than the one you are employed in for up to 30 days in any twelve-month period [1].

Cited: remote-and-hybrid-work

RetrievedPassCitedPassFactPassRefusalPassOverallPass
Is there a budget for setting up my home office?home-office · expects remote-and-hybrid-work
Answer

Yes, there is a one-off home office budget of €400 for equipment such as a desk, chair, and lighting [1]. You can claim this budget through expenses with receipts [1].

Cited: remote-and-hybrid-work

RetrievedPassCitedPassFactPassRefusalPassOverallPass
What should I do if I lose my laptop?lost-laptop · expects security-basics
Answer

You should report a lost laptop to the IT service desk within 1 hour of noticing, at any time of day [1]. The device will be locked and wiped remotely [1]. Reporting quickly is more important than knowing exactly what happened [1].

Cited: security-basics

RetrievedPassCitedPassFactPassRefusalPassOverallPass
How do I report a phishing email?phishing · expects security-basics
Answer

To report a phishing email, use the Report phish button in the mail client, which sends the message to the Security team and removes it from your inbox [1].

Cited: security-basics

RetrievedPassCitedPassFactPassRefusalPassOverallPass
After how many minutes should my screen lock?screen-lock · expects security-basics
Answer

Your screen should lock automatically after 5 minutes of inactivity [1].

Cited: security-basics

RetrievedPassCitedPassFactPassRefusalPassOverallPass
When will my laptop arrive before I start?laptop-arrival · expects onboarding
Answer

Your laptop will arrive at least 3 working days before your start date [1].

Cited: onboarding

RetrievedPassCitedPassFactPassRefusalPassOverallPass
How long is the probation period?probation · expects onboarding
Answer

The probation period is 6 months [1].

Cited: onboarding

RetrievedPassCitedPassFactPassRefusalPassOverallPass
How much does Mossgrove match into my pension?pension-match · expects benefits
Answer

Mossgrove matches your pension contributions up to 6% of your base salary [1]. If you contribute 4%, they contribute 4%, and if you contribute 8%, they contribute 6% [1].

Cited: benefits

RetrievedPassCitedPassFactPassRefusalPassOverallPass
What is my yearly learning budget?learning-budget · expects benefits
Answer

Your yearly learning budget is €1,200 [1].

Cited: benefits

RetrievedPassCitedPassFactPassRefusalPassOverallPass
Can I fly business class on a long-haul trip?long-haul-class · expects business-travel
Answer

Business class is not available on business travel, except when a medical need is confirmed by the People team [1].

Cited: business-travel

RetrievedPassCitedPassFactPassRefusalPassOverallPass
What is the hotel limit per night in London?hotel-cap · expects business-travel
Answer

The hotel limit per night in London is €160 [1].

Cited: business-travel

RetrievedPassCitedPassFactPassRefusalPassOverallPass
What is the salary range for a senior engineer?oos-salary · should be refused
Refused

Refused (model-declined). Closest document: benefits, similarity 0.54.

Retrievedn/aCitedn/aFactn/aRefusalPassOverallPass
Do I get a company car?oos-company-car · should be refused
Refused

Refused (model-declined). Closest document: expenses, similarity 0.57.

Retrievedn/aCitedn/aFactn/aRefusalPassOverallPass
Can I bring my dog to the office?oos-dogs · should be refused
Refused

Refused (model-declined). Closest document: business-travel, similarity 0.54.

Retrievedn/aCitedn/aFactn/aRefusalPassOverallPass
What was Mossgrove's revenue last year?oos-revenue · should be refused
Refused

Refused (model-declined). Closest document: expenses, similarity 0.59.

Retrievedn/aCitedn/aFactn/aRefusalPassOverallPass
Write me a short poem about autumn.oos-poem · should be refused
Refused

Refused (low-similarity). Closest document: leave-and-time-off, similarity 0.41.

Retrievedn/aCitedn/aFactn/aRefusalPassOverallPass