A diagram showing the CAM architecture.
of the captured signal data (like the temperature, humidity and pressure). Observation control The CAM allows scientists to schedule control tasks, using an observation control framework. The framework is based on schedule blocks and controlled resources. Scientists use Python scripts to control the system to do observations. Python is a general-purpose, interpreted high-level programming language that is designed to be readable.
If a scientist wants to observe a radio source, using two antennas, he or she creates a schedule block with those two antennas as controlled resources, and a reference to an observation script. Only one schedule block may use a specific controlled resource at a particular time. If the controlled resources are free (not allocated to another schedule block), then the schedule block can be activated. This causes the observation to be executed. Once the observation has run to completion, the resources are freed for use by other observations. Typical controlled resources are antennas and the correlator. Observations are scheduled for execution – put into a queue that is ordered, based on priority, required time of execution, etc. Using schedule blocks, observations can be scheduled either manually by the operator, or automatically by the CAM scheduler to execute the queued observations. System control The CAM software components are distributed over different nodes. Each
node runs a service, called a ‘node manager’, and the node manager launches the appropriate CAM components that run on that node as processes. A system controller automates and coordinates system startup and shutdown, using the ‘node manager’ services. Startup and shutdown is triggered by running a script. The start script launches the system controller as well as the components that it needs (logging component and a configuration server). The system controller coordinates startup and shutdown of CAM components via instructions to the ‘node managers’. Each ‘node manager’ controls the processes on its node by launching/stopping/(re)starting or killing components on its node as instructed by the CAM system controller or operator. The CAM uses alarms to trigger automated actions, like shutting down equipment when the cooling systems fail, or stowing the antennas when the wind becomes too strong to continue with observations. What is it made of? Architecture
The CAM is a distributed application, made up of software components, which is interfaced with client-server connection based communication via Ethernet LAN. The KATCP communication protocol is used to interconnect CAM software components, as well as connect CAM proxies to devices. KATCP is a message-based clientserver protocol on top of TCP/IP. A KATCP client connects to a KATCP server. The server exposes sensors ▲ ▲
environment, using sensors. There are 4 575 sensors in total. Physical sensors are used to monitor the environment as well as the performance of hardware, for example the temperature of the radio frequency (RF) feed of an antenna and the atmospheric humidity in the vicinity of the antennas. Software sensors are generated by software components, e.g. the disk usage of a server. Special software sensors aggregate other sensors, by applying a rule to their values. All sensors in the system are introspected at startup, and new sensors are automatically exposed, monitored and stored. Introspection means that CAM doesn’t have to know in advance which sensors are owned by the hardware it monitors – it discovers all sensors exposed by the hardware via their interfaces. Default sensor sampling is currently at 10s intervals, unless a different sampling strategy was defined. Proxies expose device sensors and log messages to ‘monitor’ components. Each ‘monitor’ component gathers the exposed sensors of specific devices. Sensor data are cached by the ‘monitor’ in memory and every 15 minutes these data are compressed to compressed CSV files and flushed to disk. About 13GB of sensor data are stored in the archive per month. Sensor data and log files stored in the archive are used for fault-finding, and to record the conditions under which observations are made. Alarms alert the user to conditions that may compromise KAT-7 equipment. A process evaluates the values of selected sensors, and generates alerts. Alerts include messages on a chat room, visual and audible indication by the graphical user interface (KatGUI), and an SMS sent to cell phones carried by support personnel. The CAM sends selected sensor values to the integrated logistic support system, which issues a work order in case of a failure, or triggers a preventive maintenance action for usage sensors (e.g. hours of operation). The CAM uses the sensor data it gathered to provide meta data (data about the captured signal data) for signal processing. For example, the metadata include information that is needed to process the captured signal data (such as where the antennas were pointing at every moment) as well as information about conditions that may be used in the interpretation
Quest 8(3) 2012 15