Community Documents
Original source document located at http://zone.ni.com/devzone/cda/epd/p/id/5997
In systems where the cost of failure is high, it is sometimes advantageous to duplicate certain critical components. This concept, known as redundancy, helps improve the overall reliability of the system by decreasing the probability of a failure. This document discusses a software method for implementing dual redundancy on a National Instruments Real-Time controller such as CompactRIO, Compact FieldPoint, or PXI. The Simple Dual Redundancy (SDR) component is set of LabVIEW VIs used to implement dual-redundant control where one controller is active and the other controller is in “hot standby” mode. The controllers communicate regularly with one another via a serial link so that if one controller fails, the other will know to take control. They also exchange information about one another’s “health” so that the active controller can determine whether or not it is in the best position to remain in charge of the system.
Implementation
The core of the SDR component is a process that runs asynchronously from the rest of the application and manages communication between the controllers. This core process is launched using VI Server and consists of a state machine in which each state for the active controller has a corresponding state for the standby controller. For example, if the active controller is sending a request for health status, the standby controller should be listening for commands. The user’s application interacts with the core redundancy process using functional (LabVIEW-2-style) global variables. Data from the API and data from the core process are synchronized during every cycle of the state machine.
Figure 1 – Software State Diagram for SDR Core
The controllers communicate with one another via a serial link, which has a low latency. The latency is also consistent since it is not affected by other traffic, as is the case with Ethernet communication. This allows a controller to quickly determine the status of the other controller so that problems can be detected as soon as possible. The amount of time in which the standby controller can detect a problem and take control is limited only by the serial polling rate. A faster polling rate translates to a quicker changeover, but requires more CPU time, leaving less processing power for other parts of the application.
The communication is message-based and is done in a command-and-response format where the active controller sends commands to which the standby controller responds. The following messages are used:
Request Health – The active controller sends this command to the stand by controller asking for its health status.
Report Health – The standby controller sends its health status as a reply to the active controller.
Request Change – The active controller sends this message when it has determined that it should hand over control to the standby controller. Immediately after sending this message it goes into standby mode.
As an added measure of error checking all serial commands include a unique “message ID” parameter in order to guarantee that a response matches a specific command. Since serial commands are buffered in a queue, if there is ever a problem with the serial link, it is possible that an old message could be received some time after it was actually sent. Including the original message ID in each reply allows the program to make sure the reply is current and relevant. For example, if the active controller receives a message that the standby controller is at 100% health, it needs to know that message was not sent 30 seconds ago.
API
SDR Start.vi initializes global variables and launches the core SDR process using VI Server. This VI can be configured to wait for synchronization to be established between the pair of controllers.
Figure 2 – SDR Start.vi
SDR Stop.vi stops the core redundancy process.
Figure 3 – SDR Stop.vi
SDR Set Status.vi sets the health status of the local controller. For example, if an I/O function returns an error, the local health can be reduced using this VI so that a changeover can happen if the other controller is in better health.
Figure 4 – SDR Set Status.vi
SDR Get Status.vi returns information about the system such as whether or not the local controller is active, whether or not the remote controller is actively communicating, the local controller’s health, and the remote controller’s health. The primary use for this VI is to determine whether or not to send outputs to the system, since only the active controller should be sending outputs.
Figure 5 – SDR Get Status.vi
Application
One of the goals of this component is to implement redundancy with minimal impact on the rest of the application. The user needs only to start and stop the redundancy engine at the beginning and end of their process, and check the operating mode (active or standby) before writing outputs.
Figure 6 – Control Program Flow With and Without Redundancy
The health parameter is optional and can be used to give a relative degree of status between the two controllers. The controller with the higher health parameter will be in the active mode. If the health parameter is not used, the active controller will remain active until a problem occurs that prevents it from communicating with the standby controller. In this mode, the communication behaves like a simple heartbeat, such that if the standby controller does not receive the heartbeat it will assume control of the system.
The core redundancy process runs at a time-critical priority, so there should not be any other VIs in the system at a time-critical priority. It is assumed that the redundancy itself is the most important piece of the system and should therefore be handled as the highest priority. This makes the SDR component ideally suited for process control applications since the standby controller can take control in less than a second should the active controller fail, effectively providing bump-less transfer for slow-speed (< 1Hz) systems.
An example program is included and can be found at <LabVIEW Directory>/examples/SDR/Simple Dual Redundancy Example.vi. The block diagram is seen in the following figure:
Figure 7 – Example Program Block Diagram
Green areas are meant to be replaced with application-specific code. The overall program flow is:
1) Start the redundancy process.
2) Read inputs from the system.
3) Perform calculations to determine output.
4) Check to see whether or not this controller is active.
5) If active, send outputs to the system.
6) Check for errors or stop condition.
7) Shut down the redundancy process.
