-
Notifications
You must be signed in to change notification settings - Fork 0
Control system operational practicalities
- Server process (acontrol) running as a plugin under the twisted python framework on the LOFAR Master Control Unit (MCU)
mcu0001. - Access to AARTFAAC station LCUs from mcu001 is over pssh for station level (rspctl) operations.
- Client processes (aclient) running as daemons monitoring a known port on:
- Correlator machines (agc001/agc002)
- Calibration pipeline machines (ais00[2-7]), a subset of which is used for processing the incoming subbands. NOTE: The calibration pipeline needs to be compiled on any of ais00[2-7], and not on ais001, since the latter has a different CPU architecture! Since /home is shared among all, a compilation on ais001 will also go through, however, the resulting binary will fail with a SIGILL on ais00[2-7].
- Atv for creating the live youTube channel.
- Communication between server and clients over TCP.
- Clients execute preformed binaries on the client machines on command from the server.
- A simple protocol is used to communicate the command line parameters between the server and the clients.
- The control server, once started, runs continuously on MCU001. On startup, it parses all the existing parset files in the folder containing lofar and aartfaac config files for scheduling observations. This is based on the
startTimefield in the parset files, and the current time. Any useful configurations are put onto a queue for scheduling with twisted. - It also monitors these locations periodically (every 10s?) for the presence of new files, and if found, parses them, checks for validity, and queues usable configurations.
- At the pre-scheduled time, acontrol initiates an AARTFAAC observation by first sending a STOP signal to all clients, followed by a START signal. These are sent out hierarchically based on a call graph, where the processes higher in the hierarchy have to successfully start before the lower level processes will start.
- If a new parset is made available during an ongoing observation, the configuration is loaded onto the queue for execution during initialisation of the next observation.
The call graph ensures that processes are started in a particular order. A process with a dependency on another will not be started if the dependency could not start up. This is implemented in acontrol/service.py:processObservation()
Every machine to be controlled by the control system needs to run an aclient with the correct parameters. Once started, the aclients run across observations, needing a restart only after a code update. Each client on every machine can then initiate an appropriate process with the correct parameters. The location of the logs of the aclients can be specified when launching the aclients. The location of the logs of the correlator can be specified on the command line, but by default is /tmp
- Create an account on portal.lofar.eu, ask Teun Grit to copy your ssh public key onto the acontrol@mcu001 account.
- Log onto
portal.lofar.euwith your username. - log onto acontrol@mcu001 for access the server code/process.
- log onto ais[0-7] from the lofar portal (These machines are now also on the LOFAR NIS).
- Note that ais001 is the head node. This implies that /home on ais001 is shared across all ais00? systems. The /data area of ais00[2-7] are mounted on ais001:/mnt/ais00? for easy access to the data products.
- Access to the station LCUs is possible via pssh from the MCU001 machine:
alias paartfaac='pssh -i -O StrictHostKeyChecking=no -O UserKnownHostsFile=/dev/null -O GlobalKnownHostsFile=/dev/null -h ~acontrol/psshaartfaac.cfg '
From the shell, commands can be run on each of the 12 AARTFAAC stations via (e.g. to check RCU status):
paartfaac /opt/lofar/bin/rspctl --rcu | grep SUCCESS -A 4
or to check subband settings on the station:
paartfaac /opt/lofar/bin/rspctl --sdo
- The control system on mcu001 is directly installed in ~/aartfaac-control in the user mode. This can be installed by
aartfaac-control$ python setup.py install --user
- On the various machines running aclients (correlators, ais00?), the policy is to create a ~/soft/src in your account for git cloning the latest code, with a ~/soft/release folder containing the build binaries.
- Thus, on client machines, cd ~/soft/src/aartfaac-control
- python setup.py install --prefix=~/soft/release/
- Make sure that the ~/soft/release is on the PYTHONPATH and PATH env. variables!
- NOTE: The calibration code needs to be compiled on ais007, which has all the development libraries installed, and not on ais001!
- NOTE: The imaging pipeline runs on
ais001. It also needs to be compiled there.
Log files are generated both by the control system processes (aclient and acontrol server), as well as by the actual binaries running during an observation. The locations of the log files are specified on the commnand line, and so can be examined via a ps -ef | grep aclient on the target machines. Log files need to be visible to all users, hence home areas are avoided.
-
acontrol on mcu001: The logs from the acontrol server are typically placed in
/opt/aartfaac/var/log/acontrol_TIMESTAMP.log. The location needs to be manually specified on the command line. -
Correlators: The correlators have a shared
/homeand/var/scratch. The logs from the aclients running on both correlators are typically placed in/var/scratch/<user>/logs/agc00[1-2]/aclient_TIMESTAMP.log. Thestderrfrom runs of the correlators are typically placed in/var/scratch/<user>/logs/agc00[1-2]/correlator/AARTFAAC-TIMESTAMP-PID.[ERROR, INFO] -
Calibration pipelines: Multiple aclients on different ports can run on a single calibration pipeline machine, with each aclient controlling an aartfaac-calibration binary operating on a single subband. The aclient logs are typically available in
ais00[2-7]:/data/logs/calibrator/aclient/aclient-TIMESTAMP-PORT.log. The application logs are typically available inais00[2-7]:/data/logs/calibrator/app/aartfaac-calibration-TIMESTAMP-PID.[INFO, ERROR]. -
Imaging pipeline: The aclient log for the imaging machine (ais001) is typically in
/data/logs/ais001/imager/aclient/aclient_imager_TIMESTAMP.log, while the imager stderr/out is in/data/logs/ais001/imager/app/aartfaac-imager-TIMESTAMP-PID.[INFO, ERROR] -
ATV: The aclient log for atv (streaming to youTube) is typically in
/data/logs/ais001/atv/aclient/aclient_atv_TIMESTAMP.log, while the atv stderr/out is in/data/logs/ais001/atv/app/<TODO>
- To kill an existing control system thread:
$ps -ef | grep twistd
acontrol 12285 25602 0 09:57 pts/7 00:00:00 grep --color=auto twistd
acontrol 26126 1 0 Apr13 ? 00:00:28 /usr/bin/python /usr/bin/twistd -l acontrol.log acontrol --config-dir /opt/aartfaac/data
$ kill -9 26126
- To run the control system, execute the following from the aartfaac-control directory on mcu001::
/usr/bin/python /usr/bin/twistd -l /opt/aartfaac/var/log/acontrol_20170706.log acontrol --config-dir /opt/aartfaac/data --lofar-dir /opt/lofar/var/run
- To examine logging messages from the control system:
tail -f /opt/aartfaac/var/log/acontrol_20170706.log
The aclients communicate with the control system server running on mcu001, accept the command line parameters, and fire off individual processes, one per aclient. Thus, an aclient needs to be running on every IP address and port specified in the "address" field of every configuration item (see below) of the AARTFAAC configuration file. All aclients need to be started manually initially, but ideally this needs to be done only once...
The aclients on the correlators can be run manually, with the location of the aclient log and the client programs' log specified on the command line. To run the correlators, the following command can be run on the correlator machines with appropriate changes:
/usr/bin/python /usr/bin/twistd --pidfile=/tmp/twistd.pid --logfile=/var/scratch/pprasad/logs/agc002/aclient_20170720.log aclient --program=correlator --numactl='-i 0-1' --logdir=/var/scratch/pprasad/logs/agc002/correlator --port=45000
The multiple aclients on the machines running the calibration pipelines (typically ais00[2-7])can be started via the following script:
The various pipelines need to be run on all the machines specified above, via a shell script similar to the one below running on each machine (aartfaac-tools/sh/start_calib_clients.sh). The number of aclients needed as per the AARTFAAC configuration file can be specified via a command line argument, e.g., bash start_calib_clients.sh 2 starts 3 clients waiting on port numbers 45000:45002.
#!/usr/bin/env bash
# Script to start aartfaac-control clients on machines running the calibration
# pipelines.
# For every machine, adjust port numbers according to the specification in
# the AARTFAAC configuration file.
# In the current form, 8 subbands are being written to a single machine.
tstamp=`date +%Y%m%d`
portstart=45000
portend=$(($portstart+$1))
for P in $(seq $portstart $portend)
do
cmd="twistd --pidfile=/tmp/aclient-$P --logfile=/data/logs/calibrator/aclient/aclient-$tstamp-$P.log aclient --program=pipeline --logdir=/data/logs/calibrator/app/ --port=$P"
eval $cmd
echo $cmd
done
The aclient running on the imaging and atv machine (typically ais001) can be run as:
/usr/bin/python /usr/bin/twistd --pidfile=/tmp/twistd_atv.pid --logfile=/data/logs/ais001/atv/aclient/aclient_atv_20170706.log aclient --program=atv --logdir=/data/logs/ais001/atv/app/ --port=45001
The above is for atv, while for the imaging pipeline, one can do:
/usr/bin/python /usr/bin/twistd --pidfile=/tmp/twistd_imager.pid --logfile=/data/logs/ais001/imager/aclient/aclient_imager_20170706.log aclient --logdir=/data/logs/ais001/imager/app/ --program=imager --port=45000
NOTE: The parameters of the aclient program (--logdir, --program and --port) need to come after the 'aclient' part of the twisted commandline. Also, run the program from within the aartfaac-control/ directory. For some mysterious reason, it does not work if executed from outside this location.
This refers to the connecting up of the various outputs of the hardware (RSP boards), correlators, calibration pipelines to each other. All of this can be controlled by the AARTFAAC configuration file.
Defined by the "starttime" keyword, this can either be "now", or in the format'YYYY-mm-dd HH:MM:SS'. This refers to the time when the AARTFAAC configuration will become active, either when the file is read (now), or at the specified time. Multiple configuration files can be present in the configuration directory, with different names (go for AARTFAAC:00000?.json), with the digits in the filename increasing in time order).
- If multiple AARTFAAC configuration files are present in the config directory, all containing the keyword "now", they will be executed in the order of appearance of the filenames in a glob pattern.
- The last valid configuration is applicable at the time a LOFAR observation is scheduled.
- Inputs: The mapping between the station UDP network outputs, and the correlator inputs is specified in the AARTFAAC configuration file. Since both correlators have to receive data from all 12 stations, this mapping is limited to which subbands are dispatched to which correlator machine. This is because agc001 receives half the subbands from all 12 stations, while agc002 receives the other half. The mapping between the subbands as reported by the rspctl output, and the ones generated as output from the correlators is described here. An example configuration snippet from the default AARTFAAC:000000.json is
"instances": [
{
"name": "agc001",
"address": "10.144.6.11:45000",
"argv": {
"i": "10.195.100.1:53269,10.195.100.1:53277,10.195.100.1:53285,10.195.100.1:53293,10.195.100.1:53301,10.195.100.1:53309"
}
Here, the "address" key specifies the location of the aclient running on the correlator machine agc001. The "i" option connects up the first half of the available subbands from the first six AARTFAAC stations to the agc001 correlator machine. In the default (reset) mode [16b, 16 subbands], these subbands correspond to Sb 313-320 for agc001, and Sb 295-302 for agc002.
-
Outputs: The correlator outputs are consumed by the calibration pipelines. Thus, the same network addresses which serve as the inputs to the calibration pipelines, are used as the output destinations for the correlators. These are specified in the "pipeline" section.
-
Logs: The Correlator output is currently written to files with the naming convention "AARTFAAC-YYYYMMDDHHMM-pid.ERROR". The path to these files can be set on the command line.
NOTE: The connection between the output subbands specified by the correlator, and the pipeline instances depends on the order with which they appear in the configuration file. E.g., if the correlators are specified in the order agc001 and agc002, followed by 16 pipeline instances, then the first 8 pipeline instances are associated with the output of the agc001, while the second 8 instances are associated with the output of agc002. This is tricky, since the actual subband number passed is only passed as a parameter to a calibration pipeline. Thus, the associated subbands should be double checked with the subbands associated with a correlator, as specified in here.
"instances": [
{
"name": "ais002-0",
"address": "10.144.6.15:45000",
"input": "10.195.100.30:4000",
"argv": {
"affinity": "2,4,6,2",
"subband": 313
}
- This snippet from the configuration file sets up the connection between the output of a correlator and the input to the calibration pipeline.
- The "address" key specifies where the client running on the ais002-0 instance is listening.
- The address in the "input" key (this is on the machine running the calibration pipeline) is used to specify both where the output of the correlator should go, and, for pipeline instances, where the input to a calibration pipeline should come from.
- The "affinity" key is used to tie the processor cores onto which this pipeline instances can be allowed to run. The first and last number in the affinity (core 2 in this case) pin the input and output thread respectively. The middle numbers pin processing threads (4 and 6).
- The "subband" key is the most important, and one needs to be sure to match it to the set subbands.