Firmware Tracks

Firmware tracks stage OPNsense updates. You sort your devices, one by one, into three tracks — fast, regular and stable — and NetDefense holds each device in a track at an exact OPNsense release. A new release reaches fast after its own soak, and reaches each later track only after it has soaked in the track before it and enough of that track’s devices have run it, or when you promote it yourself. Devices move to a release at update windows that you schedule. A device you leave out stays Unmanaged, and NetDefense changes nothing about how it updates.

The point is to let a release prove itself on a few firewalls before it reaches the rest. You might put a few low-risk firewalls in fast, a wider group in regular and your most critical sites in stable.

Tracks or the run command?

The Firmware Upgrades guide covers ndcli run firmware-upgrade, which sends an upgrade to the devices you name, now or on a schedule. Tracks work the other way round. You don’t send upgrades: you choose which track a device is in, and the track decides which release it runs and when it moves.

The two don’t mix. A device in a track, in any state and one that is still leaving it included, is refused a manual firmware upgrade. Move it to Unmanaged first, or leave its updates to the track.

What you need

• A plan that includes firmware tracks. If yours doesn’t, the Firmware page in the web app says so. See Plans.
• The NetDefense agent 1.24.0 or later on each device, and 1.25.0 or later for series upgrades. A device on an older agent can join a track, but gets no firmware step until its agent is upgraded.
• Registered devices. A device that is enabled or disabled can join a track. One still waiting for approval can’t.
• OPNsense Community edition. A device with a subscription key or a non-default release type is held, and never pinned.
• The Read-Write role to change anything. A Read-Only member, or a token that is read-only, can see everything on the Firmware page and in ndcli firmware and can change nothing.
• A schedule for each track. A track moves its devices only in the windows of the schedule you attach to it. See Schedules.

How it works

Tracks and Unmanaged

There are three tracks, always in this order: fast, regular, stable. They are created when you enable Firmware. You can edit them, but never create or delete one. Every device starts Unmanaged: in no track, and untouched.

Devices are chosen by name, one by one or several at a time. There is no OU, no pattern and no “all devices”: tracks are about individual firewalls. One request moves at most 500 devices, and it is all or nothing, so one unknown name refuses the whole request.

The pin

A device in a track is held at one exact OPNsense release, called its pin. The agent sets the pin through OPNsense’s own firmware setting, Flavour, under System > Firmware > Settings. That setting points the firewall’s OPNsense package repositories at that release’s folder on OPNsense’s mirror. While a device sits at its pin, OPNsense’s own Check for updates, Update and automatic updates see nothing newer than the pin from OPNsense’s repositories.

Joining a track pins the device at the release it runs now. Nothing is installed and nothing reboots. The agent confirms the pin, and the device moves from Enrolling to Managed. What the firewall shows, and what the pin does and doesn’t hold back, is on Firmware tracks on the firewall.

Windows

Each track runs on a named schedule, and tracks may share one. Each time the schedule fires, NetDefense sends every managed device of the track one update step. A step moves the device to the release its track has approved for the device’s OPNsense series, if that release is newer than the one the device runs. That is the highest release the track has approved that isn’t halted. If the device already runs that release or a newer one, nothing changes. A device never moves to an older release.

A step reaches a device only if all of this holds:

• The device is Managed and enabled, and its agent is 1.24.0 or later.
• No other firmware step is still open for it.
• It is online. A step is kept only for a short time, so a device that stays offline until it expires misses that window, and the next one tries again. A device that comes back in time still gets its step, and one that comes back late is held with held_late_dispatch.

A Held device gets another attempt to pin it instead of an update. A track with no schedule moves nothing, and enabling Firmware doesn’t attach one.

Whether a step reboots the device depends on one setting for the whole organization, Reboot after update, which is on by default:

• On: the full update, base and kernel included, then a reboot.
• Off: userland packages only, with no reboot. Base and kernel wait for a window that runs with the setting on.

Joining a track doesn’t update a device, but its first window can: it may be moved to a newer release than the one it runs now.

Soak and evidence

Two settings decide when a track takes a release:

• Soak is the number of days a release waits before the track takes it.
• Evidence is the number of distinct devices of the previous track that must have reached the release first.

Fast takes a release once its soak has passed. The soak is counted from OPNsense publishing the release; more exactly, from when NetDefense first sees it on OPNsense’s package mirror. Fast has no track before it, so it asks for no evidence.

Regular and stable take a release when all of this holds:

1. The previous track has approved it.
2. At least as many distinct devices of the previous track as the evidence setting asks for have reached it.
3. Its soak has passed, counted from the later of the previous track’s approval and the moment the evidence was met.
4. No unresolved failure stands against it in an earlier track.
5. It isn’t halted.

A device has reached a release when, after the release was approved in that track, its agent reports that the device runs it. A device that already runs the release counts, but its report comes with the track’s next window after the approval.

A new configuration starts with these settings:

Track	Soak	Evidence	Takes a release after
Fast	1 day	0	the soak has passed
Regular	9 days	1 device	fast approved it, 1 fast device reached it and the soak has passed
Stable	10 days	1 device	regular approved it, 1 regular device reached it and the soak has passed

You can change them: soak from 0 to 365 days, and evidence from 0 to 1000 devices. Fast’s evidence is always 0. Changing a setting never takes back an approval already given. The new values apply to decisions still to come.

Evidence needs devices in the earlier track

With the defaults, regular needs a device of fast to have run a release, and stable needs one of regular. If too few devices in the previous track can still reach a release, it stalls and waits for you. The Rollout card says Too few devices in Fast can still reach this release to meet the evidence Regular asks for. Add devices to Fast, or lower the evidence count. Add devices to the earlier track, set the evidence to 0, or promote the release.

Approval is not installation

A track approving a release only decides what its next window sends. Nothing is installed at the moment of approval. Devices move at their track’s windows, and only if the track has an enabled schedule and Firmware is enabled. The same goes for promoting a release by hand.

The states of a device

State	Meaning
Unmanaged	In no track. NetDefense changes nothing about its updates.
Enrolling	Joined the track. The agent has not confirmed the pin yet.
Managed	Pinned, and moved by the windows of its track.
Held	In the track but not pinned. The agent declined to pin the device, and says why. NetDefense asks again at the track’s next window.
Unenrolling	Leaving the track. The agent has not confirmed that the original setting is back.

A single update step can also be held. When the agent can name a precondition that isn’t met, it changes nothing and reports the reason. A Managed device stays Managed, and the next window tries again. A held step is not a failure. A failure is a step that started and did not finish, or that never answered, and it can block a release in later tracks. See Failures.

Set it up

1. Check your plan.

   Web

   Open Firmware in the sidebar. If your plan doesn’t include firmware tracks, the page says Firmware tracks are not available on your plan and you can’t enable anything.

   CLI

   ndcli org quota <org>

   The Features list shows Firmware Tracks as enabled or disabled.

2. Enable Firmware. This creates the three tracks with their default settings. No device joins a track and no track has a schedule until you set that up.

   Web

   On the Firmware page, leave Reboot after update on, or turn it off, and select Enable Firmware.

   CLI

   ndcli firmware config create

   Add --no-reboot to make the update steps apply userland packages only. You can change this later with ndcli firmware config set --reboot or --no-reboot.

3. Give each track a schedule. Create the schedule first, then attach it. Tracks may share one.

   Web

   Create the schedule on the Schedules page. Then, on each track’s card, pick it in the windows field (Fast windows, Regular windows or Stable windows) and select Attach.

   CLI

   ndcli schedule create --name saturday-night --cron "0 2 * * 6" --timezone UTC
   ndcli firmware track schedule fast saturday-night
   ndcli firmware track schedule regular saturday-night
   ndcli firmware track schedule stable saturday-night

   Attaching a schedule to a track again moves the track to the new one. ndcli firmware track schedule stable --no-schedule detaches it, and the track then stops moving its devices.

4. Add devices to the tracks. Name the devices you want in each track.

   Web

   In the Devices card, pick a track in a row’s Track dropdown. To move several devices, tick their rows, choose a track under Move to…, select Apply, and confirm the list with Move devices.

   CLI

   ndcli firmware device set fw-branch-01 fw-branch-02 --track fast
   ndcli firmware device set fw-hq-primary --track stable

   The first window can update and reboot a device

   Joining a track updates nothing, but the first window after it can. With Reboot after update on, which is the default, that window can move the device to a newer release than the one it runs and reboot it. Pick windows you can afford.

5. Watch the devices and the rollout.

   Web

   The Devices card shows each device’s track, state, pin and last step. The Rollout card shows where each OPNsense release stands in each track.

   CLI

   ndcli firmware status
   ndcli firmware rollout
   ndcli firmware history fw-branch-01

6. Tune the tracks if you need to.

   Web

   Change Soak (days) and Evidence (devices) on a track’s card and select Save settings.

   CLI

   ndcli firmware track set regular --soak-days 7 --min-success 2
   ndcli firmware track set stable --soak-days 14

   --min-success is the evidence setting.

Every command and screen is described in the web app manual and the firmware command reference.

Steer a release

You don’t have to wait for a release to work its way through the tracks.

Promote a release now

Promoting approves a release in one track, or in all three at once, without waiting for its soak or its evidence. It is the way an urgent security release goes out.

Web

On the Rollout card, select Promote… on a release, choose a track or All three tracks, and select Promote now. For a release the board doesn’t list, use Act on a release and type the release’s directory name as OPNsense’s mirror spells it, such as 26.7.5 or 26.7.3_11.

CLI

ndcli firmware version promote 26.7.5 --track all
ndcli firmware version promote 26.7.5 --track fast

Promoting always asks for confirmation. --force skips the question, and is required when there’s no terminal.

Promoting doesn’t install anything. Devices receive the release at their track’s next window, and only in tracks whose window runs. It is idempotent. The release must be one NetDefense already knows from OPNsense’s mirror, it must not be halted, and Firmware must be enabled.

Halt a release

Halting stops a release everywhere. No track approves it, and the steps queued for it that haven’t been sent yet are cancelled. Devices already on the release stay on it: there is no rollback. A halt covers the whole organization, in every track. It needs a reason, and only a release NetDefense already knows can be halted.

Web

Select Halt… on a release, type a Reason and select Halt release. Lift halt on the Rollout card brings the release back into competition for the tracks, under the soak and evidence they ask for.

CLI

ndcli firmware version halt 26.7.5 --reason "regression in the VPN stack"
ndcli firmware version halts
ndcli firmware version unhalt 26.7.5

halts lists the halts in force, newest first. Add --all to include the ones that have been lifted.

A halted release can’t be promoted until you lift the halt. The steps a halt cancelled are not restored: the next window sends whatever the track approves by then.

Failures that block a release

A step that fails, or that never gets an answer from the device, blocks the release it was moving the device to. The release can’t reach the later tracks, for the whole organization, until the same device later reaches it or you clear the failure. A failure in stable blocks nothing.

Only a failed update step counts. A hold, a step that found nothing to do, a step that expired because the device was offline, and a failed attempt to join or leave a track don’t.

Web

The Rollout card lists them under Failures blocking a release. Select Clear failures once you have dealt with the cause.

CLI

ndcli firmware failure clear 26.7.5
ndcli firmware failure clear 26.7.5 --track fast

Without --track, the failures of every track are cleared. The records stay in each device’s history.

Take devices out of a track

Moving a device to Unmanaged makes it leave its track. The agent puts back the firmware setting the device had before it joined, and the device is unmanaged again. It keeps the release it runs now: leaving a track never downgrades a device. A device that is offline or disabled keeps its pin until the step reaches it.

Web

Pick Unmanaged in the device’s Track dropdown, and confirm Release device.

CLI

ndcli firmware device set fw-branch-01 --track unmanaged

Moving a device between tracks needs no step: the pin stays where it is, and the device follows its new track’s windows from then on.

Resetting the Flavour setting on the firewall by hand doesn’t take a device out of its track. The agent puts the pin back.

Disable or delete the configuration

Disabling Firmware is not a pause. It makes every device in a track start leaving it. Each agent restores the setting its device had, and the device becomes unmanaged. Devices keep the release they run, and enabling Firmware again doesn’t bring any of them back: add them to a track again. To stop updates without taking devices out of their tracks, halt the release, or detach the track’s schedule.

Deleting the configuration is refused while any device is in a track or still leaving one. Disable Firmware or move the devices to Unmanaged, and delete the configuration once they have left. It removes the three tracks, the tracks’ windows on their schedules and the approvals. Halts and each device’s history stay, and so do your named schedules: they remain on the Schedules page.

Web

Use the switch beside the Enabled badge to disable Firmware, and Delete configuration to delete it. Both ask first. The delete dialog says the tracks go “with their schedules”, but only the tracks’ windows on your schedules are removed. The schedules stay.

CLI

ndcli firmware config disable
ndcli firmware config enable
ndcli firmware config delete

disable and delete ask for confirmation. --force skips the question, and is required when there’s no terminal.

If you remove a device from NetDefense, its pin is released as part of the cleanup, and its track membership and history go with it. See Remove a Device.

Plans

Firmware tracks are a plan feature, and the commands and the Firmware page are always there. The server decides.

• Your plan doesn’t include firmware tracks, and you have no firmware configuration. The Firmware page says Firmware tracks are not available on your plan and offers View plans. ndcli org quota <org> shows Firmware Tracks as disabled, and every ndcli firmware command, reads included, answers [FEATURE_NOT_AVAILABLE] Firmware tracks are not available on your current plan.
• Your plan stopped including them after you set Firmware up. The configuration, the tracks and the devices stay visible, read-only. No new steps are sent, and devices still in a track are being released. Four changes still work: detaching a track’s schedule, moving devices to Unmanaged, disabling Firmware and deleting its configuration. Every other change is refused.

Limits

Nothing rolls a device back. Moving a device out of a track, halting a release and disabling Firmware all leave devices on the release they run. A series upgrade has no automatic rollback either.

The pin is a guardrail, not a lock.

• It holds back OPNsense’s own repositories only. The NetDefense repository, plugin repositories and any other repository you enabled aren’t pinned, and still update.
• A release’s folder on OPNsense’s mirror can take hotfix revisions, so a pinned device can still be offered a newer revision of the release it is pinned to.
• A series upgrade that you start by hand, from the OPNsense interface, the console or a cron job, isn’t blocked. OPNsense still offers the next series on the last release of the old one, whatever the pin says.
• A device that is still Enrolling, or Held, isn’t pinned yet.

A device that stays out of contact for 30 days leaves its track. After 30 days without contact from an enabled device, the agent removes its own pin and NetDefense moves the device to Unmanaged. A disabled device counts as out of contact. Neither side sends a message. You add the device back to a track when you want it there.

Some devices can’t be pinned. A device with a subscription or a non-default release type is held. So is a device that already has a custom Flavour of its own, until you reset it to (default).

The history is recent history. A step that ended without an answer, such as one that expired because the device was offline, leaves no entry.

High-availability pairs

Each node of a pair is a device of its own, with its own pin. OPNsense’s configuration sync between the nodes doesn’t carry firmware settings. NetDefense doesn’t look for pairs when it sends a window’s update step, and it doesn’t order the two nodes. Two nodes in the same track are sent their step by the same window, and with Reboot after update on, both can restart at once.

Put both nodes of a pair in the same track and on the same schedule. The pin isn’t synced between the nodes, so the track is what keeps them on one OPNsense release, and OPNsense advises running both nodes on the same version to prevent state synchronization issues. The price is that both nodes update in the same window and, with Reboot after update on, can restart together, so the pair can be down at once. Nodes in tracks on different schedules receive each release at different times, and can run different releases for days.

Series upgrades are held for pair members. A pair is upgraded to a new series by hand, from the OPNsense interface or the console. See Series Upgrades.

Troubleshooting

Every step ends with an outcome and, when it didn’t simply succeed, a reason. The web app shows them in the Last step and Notes columns and on the device page. ndcli firmware status shows the hold reason, ndcli firmware history <device> shows each step, and ndcli task describe <code> shows the result of a step. In -f simple and -f detailed output, the reason’s code follows its sentence.

Reason as shown	Code	What it means
The firmware check failed or could not be read, or the mirror could not be reached; nothing was changed.	held_check_failed	The agent couldn’t get a reliable reading. Nothing changed, and the next window tries again. Check that the firewall can reach its mirror, and that the agent’s OPNsense API credentials are set up under Services > NetDefense > Settings.
The release folder is not on the device’s mirror, so nothing was changed. A pinned device is released if the folder is still missing at a later check.	held_release_folder_missing	The device’s mirror doesn’t carry the release’s folder. If you use a custom mirror, check that it does.
The release folder stayed missing; the device was released and is now unmanaged.	release_folder_unavailable	The folder was still missing at a second check, so the agent restored the original setting. Add the device to a track again once its mirror serves the folder.
The device has a subscription or a non-default edition.	held_unsupported_edition	Firmware tracks support the Community edition only. Nothing is pinned.
A custom firmware flavour is already set on the device.	held_custom_flavour	The agent won’t pin over a Flavour of its own. Set Flavour back to (default) under System > Firmware > Settings, and the agent tries again at the track’s next window.
Another firmware operation was running on the device.	held_firmware_busy	OPNsense runs one firmware job at a time. The next window tries again.
A reboot or a pending update is queued on the device.	held_reboot_pending	The agent won’t update on top of a queued change. Check System > Firmware on the device for a pending update or reboot.
The update plan would downgrade a package.	held_downgrade_risk	The check at the target release would downgrade a package, so nothing was changed.
The update plan was not what the step expects.	held_unexpected_plan	What OPNsense’s check lists isn’t what the track expected, so nothing was changed.
The step reached the device with less time left than an update needs, so the agent changed nothing. The next window tries again.	held_late_dispatch	Typical for a device that was offline at its window and reconnected late.
The agent is in test mode; it changed nothing.	held_test_mode	Seen only on a device whose agent was configured for test mode by hand. The device stays pinned and managed.
The mirror the device uses does not offer the release yet.	target_not_yet_offered	Nothing to do yet. For a series upgrade it can also mean the device is below the last release of its series.
The track has no release for this OPNsense series.	series_not_approved	The track hasn’t approved a release for the device’s series. Wait for one, or promote one.
The pin was changed outside NetDefense; it was put back at the installed release.	pin_drift	Something changed the Flavour by hand, or restored an older configuration. To take a device out of its track, move it to Unmanaged.
The device had no contact for 30 days, so it was released.	dead_man	See Limits. Add the device back to a track.
The device stays Enrolling, and no reason is shown.		It is offline, it is disabled, or its agent is older than 1.24.0. The Notes column of the web app and the notes in ndcli firmware status say which. An offline device is pinned once it connects, and a disabled one once it is enabled. An older agent needs upgrading.

A step that ends Failed carries the agent’s own plain-text message, for example that the update ended with a release below the target. A step that ends No answer from the device never got a reply, for instance because the device lost power or its connection during the update. Both can block a release in the later tracks until you clear the failure once you have dealt with the cause.

The reasons that belong to series upgrades are on the Series Upgrades page.

See also

• Series Upgrades — moving a track’s devices to the next OPNsense series, and its limits.
• Firmware tracks on the firewall — what the pin looks like in OPNsense, and what the agent does on the device.
• Firmware — the Firmware page in the web app.
• ndcli firmware — the command reference, including the MCP tools.
• Firmware Upgrades — the run firmware-upgrade command.
• Schedules — the cadences that drive a track’s windows.