How we test SafetyLM
| Suite | What it tests | Result | Run |
|---|---|---|---|
| Answer quality | |||
| Practitioner questions | 38 everyday WHS questions with a known answer, on the default model. 24 to 26 across runs. | 25 of 38 | Oct 2026 |
| Without the knowledge base | The same 38 questions answered from the practitioner's skill alone. What retrieval adds: 8 more correct answers. | 17 of 38 | Oct 2026 |
| Probes | Failures seen live in September, checked that they no longer happen: s 27, the WA penalty refusal, a two-turn bypass, provenance, a SWMS review and more. The one still failing: a SWMS review that doesn't name a missing rescue plan. | 7 of 8 | Oct 2026 |
| Reader roles | Eight questions each for an Advisor, Manager, Executive and Auditor, graded on the shape of the answer. | 27 of 32 | Oct 2026 |
| Searching the web | |||
| Knowing when to search | Eight questions that need current information and eight that don't. A pass is searching only when it should. | 16 of 16 | Oct 2026 |
| Current legal detail | Eight questions the knowledge base can't settle, such as a status, a commencement or a regulator's current position. Graded on an answer with a web source listed. | 8 of 8 | Oct 2026 |
| Speed | |||
| A typical answer | Median time for an answer that doesn't search the web, measured in production. | 2.3 s | Oct 2026 |
| An answer that searches | Median time when the answer searches the web. | 14.4 s | Oct 2026 |
| Computed safeguards | |||
| The penalty refusal | Penalty-figure questions for WA, SA, TAS, ACT and NZ, where no verified figure is held. Declined before any model runs, in every mode, and tested before every release. | Every time | Oct 2026 |
| Dated penalty figures | Penalty unit values for the Commonwealth, NSW, Victoria, Queensland (two) and the Northern Territory, each verified as at 1 October 2026. One more than six months old fails our release checks. | 6 held | Oct 2026 |
| The citation check | Every section, regulation and clause an answer cites, matched against the passages it drew on, up to ten in a single list. | Every answer | Oct 2026 |
| Knowledge base | |||
| Passages indexed | From 338 documents and about 2.9 million words. | 30,059 | Oct 2026 |
| Sources registered | Public regulator and industry sources in our register, 490 of them held. | 820 | Sep 2026 |
| Build quality | |||
| Automated tests | Tests that must pass before any release. | 1,800+ | Oct 2026 |
| Accessibility checks | Screen states captured at desktop and phone sizes, light and dark, with no serious or critical axe violations. | 448 | Oct 2026 |
What we changed because of them
In August and September we measured the same thing three times. A written instruction is weaker than a computed gate. Asked for a Western Australian penalty without its gate, a model said $500,000, then $10 million. With the gate, the question is declined before any model runs, every time.
So the rules that matter most are computed, not requested: the penalty refusal and the citation check sit outside the model.
What the numbers don't mean
A passing case means SafetyLM gave the expected answer to that question in that run. It doesn't mean every answer is right. Several suites grade the shape of an answer rather than its legal accuracy, and we say so beside them.
Check anything you act on against your regulator's current publication.
See how SafetyLM fits the way you work.
SafetyLM is opening in small groups. Join the waitlist and we’ll email you when your place comes up.