Begin with the intended decision
A test programme should start with an operational claim, not a marketing percentage. Identify the event the system is expected to detect, the environments in which it must work and the output an official will receive. A ball-location tool and a contact-detection tool need different reference methods. Passing a test for one purpose does not establish suitability for another, even when both tools use machine learning.
Define acceptable error, delivery time, availability and the response to uncertain input. These requirements need to reflect the sporting process. A correct output arriving after the relevant decision window may have little operational value. A fast but misleading output may be worse than a clear declaration that no reliable result is available. Officials should help define these requirements rather than receive them only after procurement.
Use a defensible reference
Accuracy must be assessed against something. The reference might be a surveyed position, a controlled timing event or a carefully established record from independent equipment. That reference has its own uncertainty, which should be considered when interpreting small differences. If two systems disagree, it is not enough to assume the more expensive system is correct or to select whichever result matches expectation.
ISO/IEC 17025 addresses the competence of testing and calibration laboratories. Its relevance is to the quality and scope of laboratory work; it is not a universal certificate that every sporting decision made with a tested product is correct. Ask what was tested, under which method and with what stated uncertainty. Accreditation or approval should be described within its actual scope.
Test difficult and representative conditions
Testing only an unobstructed object under uniform lighting can conceal the circumstances that matter in competition. Include crowding, partial occlusion, changing light, relevant weather, fast motion and challenging positions within the permitted operating environment. Different ball designs, uniforms or venue geometry may matter. The aim is not to demand performance outside the product’s scope, but to ensure the stated scope resembles real use.
For computer-vision models, distinguish development data from independent evaluation data. Reusing familiar sequences can make performance look stronger than it is. Report important categories separately, including situations where the system cannot return an answer. A high average can conceal a weak result on the rare but decisive cases officials most need help with.
Checkpoint: ask about missing outputs
Was an unresolved incident counted as an error, excluded from the reported figure or reported separately? Without that information, two accuracy claims may describe very different levels of service.
Inspect the installed pathway
Factory testing does not replace venue acceptance. Check camera positions, calibration, clock alignment, communication links and the display or receiver used by the official. Follow a test event from acquisition to receipt. Equipment may be individually functional while a network configuration or interface problem prevents the right information reaching the right person.
Repeat relevant checks after changes to mounts, software, playing surfaces or operating configuration. Keep version records so a later incident can be associated with the actual installation used. Organisers should know which changes require reassessment rather than assuming that an earlier approval automatically covers every update. A clear change-control process is part of reliability, not paperwork separate from it.
| Claim presented | Question to ask |
|---|---|
| High accuracy | For which events, against which reference, and with unresolved cases handled how? |
| Low latency | Measured from acquisition to the official’s usable output, or only within one component? |
| Approved equipment | Approved by whom, for which purpose, configuration and competition requirements? |
Rehearse failure and human interaction
Operators and officials should practise missing feeds, contradictory indications, communication loss and delayed outputs. The interface must distinguish a negative finding from unavailable evidence. An official should not need to infer system health from silence. Rehearsals also reveal whether a technically correct alert is understandable under noise, time pressure and the competing demands of live play.
Human factors matter because repeated exposure to apparently dependable outputs can encourage automatic acceptance. Training should include examples where the system is limited and where an official must seek additional evidence. This is not an argument against automation. It is a requirement that automation’s role remain legible, including the points at which the authorised person must exercise independent judgement.
Learn from incidents without promising perfection
A post-event review should distinguish a measurement error, a presentation problem, a protocol mistake and a legitimate judgement dispute. Each requires a different remedy. Preserve appropriate technical records while respecting privacy and access controls. Publish useful explanations where permitted, but avoid disclosing sensitive operational details merely to create an appearance of transparency.
No installation can honestly be described as beyond error. The stronger standard is accountable performance: justified claims, relevant testing, visible limitations and a rehearsed fallback. Readers should be able to understand not just why assistance is useful, but also what the process does when the assistance cannot answer the question.
Primary reading
- ISO/IEC 17025: testing and calibration laboratories
- FIFA: football technology programmes
- Hawk-Eye Innovations: supplier information
- Badminton World Federation: statutes and regulations
Supplier material explains supplier claims; it is not a substitute for independent evaluation or competition approval.