Versions Compared

Old Version 4

changes.mady.by.user Saeid Zoonemat Kermani

Saved on

compared with

New Version Current

changes.mady.by.user Saeid Zoonemat Kermani

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The PDU connection architecture has changed and the following is now the operating procedure for all the Eaton PDUs at Happy Jack. It is important to understand the role that joe plays in operating the PDUs so that it will be easier to trouble shoot if there are issues.

When joe starts, one of its jobs is setting up a timer to check the status of the socket connections every 60 seconds. Strictly speaking this only applies to the OMS Cards but there is a good side effect for the PDUs which is getting the status of the PDU every 60 seconds. The timer is Under the new architecture, the PDU Control view on JOE Client and the PDU widget on DeVeny LOUI do not require a login. JOE Client and DeVeny can both operate on the PDU nearly simultaneously. Each command will command does the following set of operations:

  • opens a connection
  • send the command to the PDU
  • start a readout loop and get the result back
  • close the connection.

Each command is automatically surrounded with login and logout commands so in effect, a series of commands are issued to the PDU. When JOE Client starts, it looks at the config file and creates a PDU widget for each PDU defined in the config file. We currently have 3 PDUs defined in the config file for the production environment:

Image Added 

When the view is initially instantiated, the labels for the buttons will be OUTLET-1 through OUTLET-8. As soon as the first status message comes back from the PDU, the buttons will be updated with the correct value an the timestamp on the widget will also be updated. Blue background on a button indicates that the outlet is ON. By default all the buttons are disabled for safety. The Enable button will activate the widget and one can then control the buttons. Be very careful with "Seq. OFF" and "All OFF" buttons.

The DeVeny PDU widget is very similar to the version on JOE Client except that it only shows the relevant outlets:

Image Added

One needs to be a bit patient when operating the lamps. Each action is a separate process and takes some time to make the connection, do the readout and close the connection. The PDUs by design only allow one connection at a time so activating buttons too quickly will result in trying to make a connection while the previous connection has not been terminated yet. We have not implemented a queue for the commands yet so such an occurrence will cause an exception which means that the second command will be ignored.

Generally speaking this works well but we still have to be aware of two issues:

  1. Interacting with the PDU very quickly can cause dropped commands as mentioned previously
  2. The PDU widgets do not interact well with the PDU's web interface

PDU Web Interface

The PDUs have a built in web interface which is a java applet connecting to the PDU. There are bookmarks for the PDUs on the safari browser on hj-obs2. Logging into the web interface for a PDU brings up one window that contains buttons for controlling the outlets and a series of buttons for administration such as naming the outlets. Please note

From DCT Science Report 20150415UT-C:

RC1/RC2/CAT/JOE/DeVeny:
1) 03:48    ScienceNote                  Just before Audrey was done with sky flats, Peter sent an email saying that it looked like some of the arc lamps were still on. We checked into this by taking some images on DeVeny & indeed it seemed that one was still on somehow even after I had gotten some flat images after doing some testing with the PDU connections earlier. I used the Joe Client PDU connection & the get status button didn't do anything, so I wasn't sure which lamp(s) was/were on. Without really thinking about the other outlets & the chiller being connected to one of them(they are also not labelled), I figured the easiest way to turn them off was the All off button(all while wanting to start the pointing check to get further evidence for or against an tangent pin being broken). Hence, the chiller was turned off & it didn't occur to me until Peter reminded me of the DeVeny CCD temp while I was staring at the PDU tab on Joe Client what had happened with Stephen on the phone as well.
I turned all the outlets on at this point & the ccd cooled back down nicely. After a Joe restart we took more images & used the Deveny Loui PDU control to turn the lamps off & make sure they were off.

It seems that we are having lots of problems with the DeVeny PDU and there is some confusion about how things work so I am going to write this rather long email here explaining the way it currently works.

When joe starts, it looks at the config file and tries to make a socket connection to any PDUs that are defined in the config file. Currently we only have the DeVeny PDU in the config file. joe's attempt at the socket connection will fail if the web interface to the PDU is up and running. The PDU allows one socket connection at a time.
So if the web interface is up and one wants to instead use the LOUIs for PDU communication, the web interface needs to be closed. It usually takes 30 seconds for the socket to be released so one should wait at least 30 seconds before starting joe. joe makes a socket connection but there is no login to the PDU yet. One can either login from the DeVeny or JOE Client not both. One login is allowed at a time. The login has to be successful before the button labels are populated with the right labels. If the login is not successful, the buttons look to be enabled but they will not be populated with the right labels. In other words if you don't see the labels correctly, there is no real communication with the PDU and that specific widget is not operational.
There is a timestamp at the bottom of the widget and if it turns red, it means that joe has lost the socket connection to the PDU. But being green does not mean that the widget can control the PDU. I am in the process of redesigning the code so it will be more robust and also potentially allow multiple points of access or monitoring of the PDU status. This will hopefully be ready by the next engineering run.

...

that the buttons on the web interface do not show the ON/OFF state. In order the see the socket state, you need to bring up the "Text Interface" first and then click the "Get Status" button.

...

In the case below, you see that the 4 arc lamps are off and all the other

...

outlets are on.

I then closed Using the web interface but quitting safari and making sure that the applet is not running either. I waited ~ 30 seconds and brought up joe and joe Client and this is what I saw after logging in:
Image Removed
You see that the state of the buttons are obvious by whether the button has a blue background or not and that all the labels are updated. If the buttons had shown OUTLET-1 through OUTLET-8 as the label, it would have meant that I do not have a good connection. At this point, I did not bother with the DeVeny PDU because I knew that JOE Client would have pre-empted any other connectionsis not recommended and is a last resort. It is also essential to make sure that the browser window is close when it is not needed any more, otherwise the PDU widgets on different LOUIs cannot make a connection to the PDU.