FIES CCD3 known problems and their suggested solution

When using multiple observing systems

When observing, only have ONE observing system started (running) at any time.
The basic rule is to use 'shutdownobssys' and 'startobssys' for stopping and starting observing systems.

1. Open an X-term on 'selena' and type
   'stop-ccd3-fies'


2. Start the ALFOSC observing system, using 'startobssys alfosc'

1. Open an X-term on 'selena' and type
   'stop-ccd3-alfosc'


2. Open an X-term on 'selena' and type
   'start-ccd3-fies'

1. Open an X-term on 'selena' and type
   'stop-ccd3-fies'


2. Open an X-term on 'selena' and type
   'start-ccd3-alfosc'

General recommendations

In case you get a problem with the FIES observing system, this is the recommended order of remedies to try before calling the technical support staff:

1. Run the offending command or script again.
   If that doesn't work then


2. Run the command 'restart-ccd3-fies' (on selena, any terminal window). This will restart the ccd3comm program and the database daemon. Look at the "Status Delay" in the CCD3 Status Display. It will count up to about 7s, and the system is ready again. Consider waiting until the on-going exposure is finished before doing this, as any exposure will be lost. If this dosen't work then


3. Shut down the observing system and start it again.
   Doing this up to three times can get things going again. If that doesn't work then


4. Shut down the observing system, log out of Lisa, log in again and start the observing system. If that doesn't work then


5. Reboot the ccd3-fies computer. Do this by executing the command /usr/local/bin/reboot-ccd3-fies. Do ping ccd3-fies to see when it is up again, so you can start the observing system.

As a last resort, if all the above did not resolve the problems, it may be necessary to power cycle the CCD controller. The CCD controller is inside the FIES white box, but can be power-cycled remotely.

• Shut-down the FIES obssystem. Switch off the CCD controller: in a black terminal do

          [obs@selena ~]$ fies.ccdcontroller_off

  then wait 20 sec, and switch on the CCD controller with

          [obs@selena ~]$ fies.ccdcontroller_on

  and again try to start the FIES obssystem.

• If your trouble-solving brings you as far as this item, please use the 'system.speedtest' command to check for possible network issues.

Specific solutions to known problems

• If the "Status delay" number on the ccd3 status display is red and is counting up try running 'restart-ccd3-fies' once or twice.
  If it doesn't work then shut down the observing system and start it again.



---

• if you see error messages in the Talker like this:
  
  [ERROR:] Cannot create WCS information for this image (system is 2). xbin is 0. Looks like the CCD status database is not updated.
  
  Then shut down the observing system and start it again.



---

• If you see error messages in the Talker like this:
  
  Command timed out looking for CCD3 program (fies-ccd3) to talk to (bus=161.72.5.255)
  
  Then re-run the command or script that caused the error. If it doesn't fix the problem shut down the observing system and start it again.



---

• If you get errors like this:
  
  [ERROR]: Failed to connect to database: Error: User 'ccd_user' has exceeded the 'max_user_connections' resource (current value: 50)
  
  restart the fitswriter program as described here and restart the ccd3db program as described here
  
  

---

• If you notice that the readouot speed is not what you expect (FPIX=129 (or other) instead of FPIX=200), do (in this order) 'resetxy', 'readspeed what-you-want', 'resetxy'. Only after these commands should you set/change the binning if needed.



---

• When windowing the ccd, set the X and Y size (with 'xsize' or 'ysize') before setting the beginning of the window (with 'xbeg' or 'ybeg').
  If you do the reverse and start an exposure, the software will freeze when starting to read out the image. If this occurs the image is lost and you have to restart the observing system.



---

• When changing readout speed to 800 kpix/sec using the command 'readspeed 800' always remember to give the command 'ampl B' (or the equivalent command if you want to use another amplifier) afterwards.
  If you do not, the bias level will be wrong.



---

• Changing readout speed changes the overscan value from 10kADU to e.g. 12kADU, 16kADU, 23kADU or 4.294903E9ADU, depending of the combination. You can normally recover to the 10kADU level by giving command e.g. "resetxy" or "xbin 1".



---

• If you do not get the prompt back when using the exposure commands ('expose', 'dark', 'mexpose' or 'mdark') and the images are not present in the normal image directory (/data/alfosc), then there is a problem with the "remsave-client". Contact the support staff.
  Note that the images made are, in fact, being exposed and stored locally on the 'ccd3-fies' computer. They can be manually copied from 'obs@ccd3-fies/images' to 'obs@selena:/data/fies'.



---

• If the exposure meter is not working, i.e. 0 current counts and 0 accumulated counts during exposure on bright target try running 'fies.expmservers-restart'. This will restart the FIES expmeter server programs. The restarting of the servers may take up to 45 seconds. Regarding the exposure-meter servers, this command is equivalent to restarting the obssystem.
  Note that the exposuremeter will not automatically count after restarting the servers, it has to be commanded to start counting (again), e.g. with the 'expmcount' command.



---

• If the following error appears "Cannot connect to 161.72.5.1xx/port 400[1-2]", where 'xx' is the last two digits of the IP address, it is most likely that the MOXA bos for the application (FIES, StanCam, etc.) needs to be restarted. This can be done by a entering the following webpage here, copy the IP adress for offending 'Application' into a new browser, login with the obs account, and select 'save/restart' and restart it. Wait 10 s and restart the observing system.