Source: docs/operations.md

Operations runbook

How to run HerdTag tags on a real farm. This is for prototype trials. The virtual fence isn't a safety barrier. Keep animals inside a working physical perimeter fence, away from roads and hazards, for the whole trial.

Daily check (5 minutes)

Open /farmer on a phone. The headline should read "All N cows inside …".
Open / (the map). Every tag should show as synced (no "syncing" pill) and checked in within the last 15–20 minutes.
Read anything new in the events list: breach, cue_ignored, battery_low, gps_degraded.
During cattle trials, also open /trial and export the CSVs (see below).

Adding a tag

Register it in The Things Stack: Application → Register end device, frequency plan US915 FSB2, LoRaWAN 1.0.3, OTAA. Use a DevEUI/AppKey you control, and a device id like tag-11. That id is what the dashboards show. See hardware/BUILD.md §7.
Flash it with those keys in firmware/tag/secrets.h (Firmware). Check on Serial that it joins and gets a fix outdoors.
Power it on outside. The tag appears in /api/tags and on the map after its first uplink, reporting fence_id = 0 (no fence).
Give it a fence. "Send to all" on the map, or POST /api/fences/{id}/assign with {"dev_ids": ["tag-11"]}. "Send to all" only covers tags that have already uplinked, so do this after step 3.
Wait for "synced". The tag pulls one fragment per uplink, and uplinks every 30 s until the fence is complete, so this normally takes a minute or two. Don't fit it to an animal until it's synced.

Replacing a tag (lost, broken, flat)

Register and flash the new tag with a new device id, then follow "Adding a tag".
Write down which animal wears which tag id. The system has no animal records yet, so keep that mapping in your trial notes.
The old tag's row stays in the dashboards and will show as "not checking in". There's no delete endpoint yet. Ignore it, or rename it in your notes.
If the tag is lost in the field, the last known position is on the map and in /api/tags/{id}/track.

Changing a fence

Draw the new polygon on the map ("New fence"), name it, and save. The backend rejects shapes with crossing edges or more than 32 corners.
Send it to the tags. Each tag keeps enforcing its old fence until it has fully received and checked the new one, then confirms with fence_synced.
Watch the fence row's "n/N tags synced" count. Before relying on the new boundary: - make sure every tag is synced, and - make sure no animal is already outside the new boundary. An animal that's outside when the fence arrives goes straight to breach, and gets no cues.
Archive the old fence (✕) once no tag is assigned to it. The server refuses while any tag still is.

Move fences in small steps when animals are learning. A sudden jump puts animals outside the boundary, with no cues to guide them in.

What each alert means

Alert / event	What it means	What to do
Cow OUT / `breach`	Tag is confident the animal is more than 3 m outside the virtual fence. The tag has gone silent	Check the animal's position. Bring it back calmly if needed. Check whether something is drawing animals out: water, better grass, a gate. Repeated breaches by the same animal go in the trial notes
`returned`	Back inside by 5 m or more	None. Note it
Kept walking despite the warning / `cue_ignored`	10 cue bursts with no turn in one episode; the tag stopped cueing	Look at the animal and the spot. Possible reasons: it can't hear the cue (tag position, wind), it's driven by something strong, or it isn't trained yet. For welfare, repeated `cue_ignored` on one animal is a trial stop-and-review trigger
Tag not checking in (> 1 h)	No uplink for over an hour	See "tag not heard" below
Battery low / `battery_low`	≤ 20 %	Recharge or swap within a few days. The percentage is approximate
`gps_degraded`	Fix accuracy worse than 8 m. The tag won't cue until it improves	Usually tree cover, buildings, or the animal lying on the tag side. Persistent on one tag means an antenna or enclosure problem
Hasn't picked up the new fence / "syncing"	Assigned fence not yet confirmed	See "fence not syncing" below
`unknown_payload`	Uplink on the wrong port or of the wrong length	Firmware mismatch. Reflash that tag

Battery and charging

Charge with USB-C, or with the solar input (5 V panel, never over 5.5 V).
LiPo cells must not be charged below 0 °C. In winter, bring tags in to charge, or treat solar charging as off.
The battery reading is a rough linear estimate from voltage. For trials, note the voltage at fitting and at removal.
Until power is measured on the bench, treat battery-life predictions as guesses. The simulator suggests that animals spending a lot of time near the fence drain tags much faster.

Troubleshooting

A tag isn't heard:

Check in the TTS console (Live data) whether the gateway sees it at all. If TTS sees it but the backend doesn't, the webhook is the problem: check the URL, the X-Webhook-Secret, and that the server is reachable.
If TTS doesn't see it: check the gateway is online and on FSB2, and that the tag is in range (compare RSSI in /ops). Then the battery, then the antenna connection.
Power-cycle the tag outdoors and watch Serial if you have a laptop with you.

A fence isn't syncing:

Fragments only go down after the tag uplinks. A silent tag never syncs; fix that first.
With TTS configured, check the downlink queue for the device in the TTS console. In the backend, downlinks.status = error holds the TTS error text.
Re-sending the fence supersedes older queued frames, which is safe to do.
A CRC failure on the tag discards the fence quietly, and the tag keeps the old one. Repeated failures point to a firmware/protocol mismatch.

GPS degraded a lot:

Check the tag's orientation (patch antenna facing the sky) and that nothing metal is near the antenna.
A first fix after power-up needs open sky and can take a minute or more.

Positions jumping around:

Up to about 5 m of jitter is normal. Large jumps with good h_acc suggest a firmware or antenna problem; export positions and look.

Data export and backups

curl -o positions.csv 'http://127.0.0.1:8000/api/export/positions.csv?hours=744'   # last 31 days
curl -o events.csv    'http://127.0.0.1:8000/api/export/events.csv?hours=744'

Everything lives in one SQLite file (HERDTAG_DB, default herdtag.db). To back it up while the server runs:

sqlite3 herdtag.db ".backup 'herdtag-$(date +%F).db'"

Keep one daily backup off the machine during trials. Don't delete the DB mid-trial. Fence ids restart at 1 in a fresh DB, and a tag still enforcing an old fence 1 would look synced to the new server when it isn't.