Writing an Annotation Statement of Work That Holds Up in Production
Turn shared expectations into a workflow people can follow.
“Label the grasp” sounds clear until two people disagree about when it starts. “Deliver high-quality annotations” sounds reassuring until a batch arrives and nobody has agreed how to accept it.
An annotation statement of work turns those expectations into something both teams can use. The useful version describes the data, the labels, the checks and the decisions that happen when reality differs from the plan. It should help an annotator, reviewer and project lead reach the same understanding of what the next batch requires.
Define the inputs and the unit of work
Start with the source material: the types of video, expected duration, recording characteristics and the metadata supplied with it. State what happens if files are missing, corrupted or outside the agreed scope.
Define the billable unit separately from the delivery unit. A frame price, a video-hour price and an episode price can describe very different workloads. Specify whether idle time, empty frames, review and rejected inputs are included.
Give the batch a stable identity. The vendor and buyer should be able to refer to the same source files and versions without relying on a folder name that someone may later change.
Make the taxonomy operational
A list of class names is a starting point. Each class needs a definition, examples and guidance for confusing neighbouring cases.
For temporal events, state the start and end rules. For hand-object contact, explain whether the label follows visible contact, an inferred state or another defined criterion. For keypoints, state the skeleton, coordinate system, units and visibility rules.
Include a way to mark uncertainty or unusable evidence. Do not require a precise label when the video cannot support one. Explain who adjudicates the case and whether the resulting decision changes the guidelines for future work.
AWS’s instruction-writing guidance emphasizes concise explanations and examples tested on actual data. An SOW can reference a versioned guideline document so those examples remain connected to the agreed scope.
Specify outputs your pipeline can read
Name the delivery format and provide a sample schema. Define source identifiers, time units, frame indexing, coordinate conventions, class values and schema versions. Decide how the team will represent missing values rather than leaving that choice to individual annotators.
Ask for a small export before production. Run it through the receiving pipeline and record any corrections. This is a practical acceptance step: it catches structural problems before they appear across a large delivery.
Preserve the connection between labels and the exact video version. Trimming, redaction or changes in frame timing can affect that relationship, so specify how transformations are recorded and communicated.
Give every quality measure a job
Choose measures appropriate to the label type. A temporal tolerance should specify the event and reference it applies to. A class score should make clear which classes and samples it covers. A keypoint measure needs an agreed treatment of invisible or ambiguous points.
Separate consistency from correctness. Agreement between annotators can reveal unclear instructions or drift, but it does not replace comparison with an adjudicated reference. State how that reference is created, reviewed and updated.
Define the sampling plan, review responsibilities, reporting frequency and thresholds before production. Also define what happens when a small but important class has too few examples for a meaningful batch score. A single overall percentage can hide the part of the dataset you care about most.
Write down the acceptance process
The SOW should say when a batch is delivered, what the buyer checks and how findings are reported. Agree a review window and the process for accepting, rejecting or partially accepting a delivery.
Make issue reports specific enough to act on: source identifier, label layer, guideline reference and example. Decide how disputed findings are escalated and who can make the final operational decision.
Avoid treating silence as a shared understanding. If the agreement includes a consequence for a missed review deadline, both parties should understand it explicitly and have the contract reviewed accordingly.
Distinguish rework from a new request
A label that breaks the agreed rule and a label that follows yesterday’s rule after the buyer changes it are different situations. Describe how the team will identify the cause and agree responsibility, cost and timing.
Keep a change log with the taxonomy version, affected batches and approval date. Decide whether a change applies only to future work or also requires earlier labels to be updated. Do not leave that decision buried in a chat thread.
Also cover access permissions, allowed subcontracting, data handling, retention and handover at the end of the project. These operational details should align with the parties’ wider contract and security requirements.
Use the pilot to finish the specification
Treat the pilot as a chance to test the SOW in practice. Review actual disagreements, update examples and validate the export before committing to production volume.
Guardian gathers and annotates egocentric data. A useful starting conversation covers the tasks, label layers, evidence available in the footage and the checks that will make a delivery acceptable. The SOW records those decisions so the whole team can work from them.
