Watchdog and force discipline: brownfield safety habits

Watchdog and force discipline is the part of brownfield commissioning that nobody writes down because nobody wants to admit how often it goes wrong. The pattern is the same on every site: a contractor on a Friday afternoon forces an output to confirm a sensor wiring fix, gets distracted by a phone call, signs off the job sheet, drives home, and leaves the force in place. The next operator presses the stop button on Monday morning, the output stays high because the force is still active, and the conveyor that should have stopped is still running when somebody walks into the guard area to clear a jam. There is no software bug. There is a discipline failure, and the discipline that prevents it is a watchdog timer plus a force log plus a clear-on-restart routine. This tutorial walks the pattern.

Why this matters on real plants

Forces are the most powerful and most dangerous tool in the engineer's toolbox. A force overrides the program logic completely — the rung that should evaluate to false because the safety gate is open is bypassed, and the output asserts true because the engineer typed 1 into the force value field. Used for thirty seconds during a controlled wiring test on a tagged-out machine, a force is a useful diagnostic. Left in place for a week on a running line, a force is a recordable incident waiting to happen. The cost of a forgotten force scales with the energy of the actuator it was applied to — a 24V indicator lamp that stays lit costs nothing; a 22 kW conveyor motor that stays running while a fitter is in the guard zone costs everything.

We have walked into a brownfield petrochem site where the engineering laptop's force list had 47 active forces accumulated over six months, half of them on inputs from sensors that had been replaced and re-wired since. The PLC was running on a fiction — a snapshot of what the engineers thought the field looked like in November of the previous year, frozen in place by forces that nobody had cleared. The fix was straightforward but tedious: walk every force, verify the underlying field condition, release each force one at a time, and watch the program respond. It took two engineers four days. The discipline that would have prevented those four days is a five-line force watchdog plus a one-page force log.

The third reason it matters more in SA than in textbook examples: contractor turnover on most sites is high, and the engineer who applied a force in March is rarely the engineer doing the wiring fix in September. A force list with no audit trail — no who, no when, no why — is a list nobody is willing to clear because nobody knows what it does. The audit trail is what makes the force list manageable across shift changes, contract changes, and the inevitable handover gaps. Without it, the force list grows.

The mental model

Every force on a PLC sits in one of three states. State one is active and intentional — a wiring test in progress, an engineer in front of the panel with a meter in hand, a clear plan for releasing the force in the next ten minutes. State two is active and forgotten — a force applied by an engineer who has since left the panel, the laptop, or the site, with no record of why. State three is released — the force has been removed and the program logic is back in control. The discipline is the set of habits that keeps every force in state one or state three, and never lets one drift into state two.

The watchdog is the failsafe that catches the drift. A simple TON timer fed by the rising edge of the force-active bit on a critical output will time out at a preset interval — 30 seconds on the simulator example below, 5 to 30 minutes on a real plant depending on the criticality of the output. When the timer elapses, the watchdog triggers an alarm, sets a force-stale bit, and writes an entry to the force log with the elapsed time and the output address. The operator sees the alarm. The engineer sees the entry. The force gets released, or the engineer renews it explicitly with a fresh timestamp.

The force log is the second piece. Every force-set and every force-release event writes a record to a logging mechanism — a CSV on the engineering laptop, an entry in the SCADA's audit table, a row in the historian's event log. The record carries the address (Local:1:O.Data.0), the value applied (1), the user account that applied it (the Windows username on the engineering laptop, or the SCADA user who issued the command), and the timestamp. Without the user field, the audit trail is useless — you cannot ask the right person what the force was for.

The clear-on-restart routine is the third piece, and it is the most controversial. Some sites set every force to release on PLC restart, on the theory that a controlled restart is a clean slate and any force worth keeping will be reapplied deliberately by the engineer who knows it is needed. Other sites preserve forces across restarts on the theory that an unintended restart should not change the field state. The right answer depends on the plant's commissioning culture. The wrong answer is to leave the choice undocumented — every site should have a written rule, and the rule should be enforced by the PLC's startup OB / first-scan routine.

Worked example

Open the simulator. Drop a CompactLogix CPU on the rack with a DI16 input module and a DO16 output module. Load the latch ladder preset and configure a simple start-stop motor circuit on output Local:2:O.Data.0, driven by the latch logic from the ladder-logic-basics tutorial. Run the program and confirm the start button latches the motor on and the stop button drops it. This is the baseline.

Now apply a force. Open the force-edit panel in the simulator and set Local:2:O.Data.0 to 1 with force enabled. The motor turns on regardless of the start-stop logic. The simulator's force indicator lights up — a yellow F next to the output address — and the force-active bit Forces.Output_Force_Active[0] goes true. Press the stop button. Nothing happens. The output stays high because the force is overriding the program. This is the symptom that motivates the discipline: the operator's stop press has no effect, the motor is running, and there is no obvious indication on the HMI that a force is active because the standard motor-status display reads the output bit, which is exactly what the force is asserting.

Now wire the watchdog. Add a TON timer to the program with a 30-second preset, fed by the force-active bit:

(* Force watchdog — fires if any output force stays active for >30s *)

FORCE_WATCH : TON;
FORCE_LOG_TRIGGER : R_TRIG;

FORCE_WATCH(IN := Forces.Output_Force_Active[0],
            PT := T#30s);

(* Stale-force alarm latches when the timer elapses *)
IF FORCE_WATCH.Q THEN
    Alarms.Force_Stale[0].Active := TRUE;
    Alarms.Force_Stale[0].Output_Addr := 'Local:2:O.Data.0';
    Alarms.Force_Stale[0].Elapsed_Sec := TIME_TO_DINT(FORCE_WATCH.ET) / 1000;
END_IF;

(* Force-set and force-release events get logged on every transition *)
FORCE_LOG_TRIGGER(CLK := Forces.Output_Force_Active[0]);
IF FORCE_LOG_TRIGGER.Q THEN
    LogForceEvent(addr := 'Local:2:O.Data.0',
                  value := Forces.Output_Force_Value[0],
                  user := SCADA_User_ID,
                  timestamp := PLC_RTC);
END_IF;

Run the scenario. Apply the force at simulator time 00:00:00. The TON starts counting. The motor stays high because the force is asserting it. Press the stop button — nothing happens. At 00:00:30 the TON elapses and the stale-force alarm fires. A red banner appears on the alarm summary: Force_Stale: Local:2:O.Data.0 — elapsed 30s — applied by SCADA_User=engineer1. The alarm is unmissable. Even an operator who has not noticed the yellow F indicator on the output now has a high-priority alarm telling them a force is overriding the program logic.

Release the force. Open the force-edit panel, untick the force-enable checkbox on Local:2:O.Data.0, click apply. The force-active bit drops to false, the TON resets, the stale-force alarm clears, and the force-release event writes to the log. Press the stop button — the motor drops. The program logic is back in control. The audit log now has two entries: a force-set at 00:00:00 by engineer1, and a force-release at 00:00:42 by engineer1. The next engineer to look at the log can see exactly what happened, who did it, and how long the force was in place. That is the audit trail working.

Now restart the PLC. The simulator's CPU panel has a restart button — click it and the program reloads. Watch what happens to the force. With the simulator's clear-forces-on-restart flag set to TRUE (configurable in the CPU properties), the force list is empty after the restart. The motor follows the start-stop logic again. With the flag set to FALSE, the force survives — the motor is high again immediately on restart, before the program has even completed its first scan. The choice is yours; the discipline is to make the choice explicitly and document it on the project handover.

Common mistakes

• Forgetting to clear forces before leaving site. The single most common discipline failure on a brownfield wiring fix. The contractor applied a force at 14:00 to confirm a sensor connection, finished the test at 14:15, started troubleshooting an unrelated issue at 14:30, and forgot the force was still active when they signed off the job sheet at 17:00. The fix is a checklist on the job sheet — no sign-off without an explicit force-list review and a screenshot of the cleared force panel attached to the work order.

• Forcing a process variable instead of an HMI command. A common rookie mistake during a wiring test is to force the process variable (the analog input from the temperature sensor, for example) to a target value to test what the program does at that PV. The program response is then a fiction — the actuators move because the force tells them to, but the underlying physical process has not moved. If the engineer then tests the alarm trip at the forced value and the alarm trips correctly, they have proven nothing about the field — only about the program. Always force the HMI command or the engineer's manual override input, never the PV. The PV must always reflect what the field actually reads.

• Forcing in run mode without a force-release checklist. Forcing in run mode on a live machine is sometimes necessary — a wiring test on a guarded but running conveyor, for example — but it must be done with a written checklist that includes the release. The checklist starts with a force-list snapshot before the work, records every force applied during the work, and ends with a force-list snapshot after the work that confirms only the pre-existing forces remain. Without that bracket, an applied force can disappear into the existing force list and never get noticed until the watchdog catches it days later.

• No audit log of who forced what when. A force list with no who field is useless after the fact. The engineer who applied the force three months ago is now on a different site, the contract has rolled over, and nobody on the current shift knows what the force was for. The audit log must include the user account that applied the force, the timestamp, the output address, and the value. Most modern PLCs (CompactLogix, S7-1500, Modicon M580) provide hooks for this in the firmware audit trail. If your PLC does not, write the equivalent logic in the program — the five-line code block above is a starting point.

• Trusting the engineering laptop's force panel as the source of truth. The engineering laptop shows the forces it has cached locally. The PLC is the source of truth. A force applied via SCADA, via a different engineering laptop, or directly through the CPU's web interface will not appear on the local laptop's force list until it does a full upload. Always check the force count on the CPU diagnostic page (F count on the Siemens display, force-list summary on the Allen-Bradley diagnostic) before claiming the program is force-free.

• No watchdog on safety-related outputs. Forces on safety-related outputs (a guard-door release, an emergency-stop bypass, an interlock disable) need a much shorter watchdog than forces on a status indicator. A 30-minute watchdog on a guard-door bypass force is a 30-minute window for the bypass to be forgotten while the door is still bypassed. Set the watchdog on safety-related outputs to a maximum of 60 seconds, and make the alarm priority critical. The engineer who is genuinely doing safety work for longer than 60 seconds can renew the force; the engineer who has wandered off cannot.

How to practise this in the simulator

The simulator's latch preset and the force-edit panel are wired together for this pattern. Open the start-stop motor program, force the output high, watch the stop button do nothing, and watch the TON force watchdog count up to 30 seconds and trip the alarm. Then release the force and watch the audit log update. Switch the clear-on-restart flag and watch what happens to the force across a CPU restart. Try forcing the start input low while the motor is running and watch the program logic ignore the stop button — same symptom from a different cause, same watchdog trips, same audit entry. Twenty minutes of breaking and fixing teaches more about watchdog and force discipline than any procedural document.

Vendor reference

The cross-vendor reference for force handling and audit trails on the Allen-Bradley platform is the Rockwell Automation Support knowledge base, which has detailed articles on the FORCE instruction, the force-list export, the FactoryTalk audit log, and the firmware-level audit trail on ControlLogix and CompactLogix CPUs. The Siemens equivalent on TIA Portal lives in the online documentation under "Modify variables" and the S7-1500 audit log feature. Both vendors recommend the same pattern: a force is a temporary diagnostic, not a programming construct, and every applied force should be logged and reviewed.

What we don't claim

This site is not SAQA-registered, not MerSETA-accredited, and not an NQF-registered qualification provider. Our completion certificates are course-level only — they describe what you covered, not an NQF Level X qualification. The CCST cert from ISA is the portable industry credential we recommend; we are not an ISA cert delivery partner either, but our cert packs are CCST-aligned. Force discipline is one of the topics where the cert syllabus and the on-site reality diverge — the syllabus covers the FORCE instruction; the simulator practice teaches the discipline that keeps it from killing somebody.