Operating a wide-area remote observing system for the W. M. Keck Observatory Gregory D. Wirtha , Robert I. Kibrickb , Robert W. Goodricha , and James E. Lykea a W.
b UCO/Lick
M. Keck Observatory, Kamuela, HI 96743 USA Observatory, University of California, Santa Cruz, CA 95064 USA ABSTRACT
For over a decade, the W. M. Keck Observatory’s two 10-meter telescopes have been operated remotely from its Waimea headquarters. Over the last 6 years, WMKO remote observing has expanded to allow teams at dedicated sites in California to observe either in collaboration with colleagues in Waimea or entirely from the U.S. mainland. Once an experimental effort, the Observatory’s mainland observing capability is now fully operational, supported on all science instruments (except the interferometer) and regularly used by astronomers at eight mainland sites. Establishing a convenient and secure observing capability from those sites required careful planning to ensure that they are properly equipped and configured. It also entailed a significant investment in hardware and software, including both custom scripts to simplify launching the instrument interface at remote sites and automated routers employing ISDN backup lines to ensure continuation of observing during Internet outages. Observers often wait until shortly before their runs to request use of the mainland facilities. Scheduling these requests and ensuring proper system operation prior to observing requires close coordination between personnel at WMKO and the mainland sites. An established protocol for approving requests and carrying out pre-run checkout has proven useful in ensuring success. The Observatory anticipates enhancing and expanding its remote observing system. Future plans include deploying dedicated summit computers for running VNC server software, implementing a web-based tracking system for mainlandbased observing requests, expanding the system to additional mainland sites, and converting to full-time VNC operation for all instruments. Keywords: remote observing, VNC protocol, observation scheduling
1. INTRODUCTION We provide here a brief overview of the Observatory’s remote observing facilities in Waimea and California.
1.1 Remote Observing from WMKO Headquarters in Waimea Most observations with the Keck telescopes are now carried out from one of the two remote operations rooms located at WMKO Headquarters in Waimea (see Fig. 1). The workstations in these rooms communicate with instrument and telescope control computers at the summit via a dedicated, 1 Gbit/sec fiber link (See Fig. 3). The observer is supported by an observing assistant (OA), who operates the telescope, and a support astronomer (SA), who assists the observer in setting up and operating the instrument. During the first part of the night, an SA is present in the remote operations room and during the latter part an on-call SA can be reached by telephone at home. The OA is usually located at the summit, but in some cases will operate the telescope from the same control room in Waimea from which the observer is running the instrument. A video-conferencing system links each remote operations room in Waimea with its corresponding telescope control room at the summit. Further author information- (Send correspondence to G.W.) G.W: Email:
[email protected]; http://www2.keck.hawaii.edu/realpublic/inst/people/wirth/; Telephone: 808-885-7887; R.K.: Email:
[email protected]; http://www.ucolick.org/˜kibrick; Telephone: 831-459-2262; Fax: 831-459-2298; R.G.: Email:
[email protected]; Telephone: 808-885-7887; J.L: Email:
[email protected]; Telephone: 808-885-7887
Figure 1. WMKO remote operations room in Waimea
Figure 2. Santa Cruz remote observing room
1.2 The WMKO Remote Observing Facilities in California The remote observing facilities in California (e.g., see Fig. 2) are primarily targeted towards observers who live within commuting distance of one of those facilities.1 They are not intended to duplicate the Waimea facility nor to operate independently of it. Rather, each is an extension of the facility in Waimea. That facility and those on the mainland are intended to operate in collaboration, sharing resources where practical. We rely on the existing instrument support staff at Waimea and provide video-conferencing and shared software environments so that they can most effectively support the observers at the mainland sites.
Mainland Site
WMKO Summit (Mauna Kea)
WMKO Headquarters (Waimea)
Astronomer
Observing Assistant
Support Astronomer
X display
Video conference
X display
Video conference
X display
W/S
W/S
W/S
(Internet−2) F
R
R F
T
T
(ISDN)
Video conference
(Fiber) R
R
Legend Computer control
Computer control
Telescope
Instrument
F = Firewall R = Router T = Telephone co. switch W/S = Workstation ISDN circuit
Figure 3. Global overview of connectivity between mainland remote observing site, Mauna Kea, and Waimea.
Communication between the sites is provided via the Internet-2 network (see Fig. 3). However, most California remote observing sites are also equipped with a series of dialed ISDN circuits that provide an alternate link to the Mauna Kea summit. In the event of a loss of Internet connectivity, remote observing traffic automatically fails over onto this ISDN backup path, and automatically falls back once Internet connectivity is restored.2
1.3 History Initial development efforts were undertaken by WMKO staff and were primarily driven by the goal of moving observers off of the oxygen-starved Mauna Kea summit and enabling them to observe with greater safety and efficiency from a nearby facility located at a lower altitude.3 The Keck I remote operations room in Waimea became operational in 1996, and the corresponding room for Keck II (see Fig. 1) came online the following year. Subsequent efforts to extend this remote observing capability to mainland sites were undertaken by researchers at the University of California, Santa Cruz (UCSC)4 and the California Institute of Technology (CIT).5 A key motivation for those efforts was to reduce the travel time and costs associated with short-duration observing runs.1 The remote observing facility at UCSC became fully-operational in 2001, and served as the model for similar facilities that came online at CIT and at other UC campuses during the next seven years (see Table 1). Table 1. WMKO remote observing sites
Number
Name
Location
First use
1
Remote ops 1
Waimea, Hawaii
1996
2
Remote ops 2
Waimea, Hawaii
1997
3
UCSC
Santa Cruz, California
2001
4
UCSD
San Diego, California
2003
5
CIT
Pasadena, California
2004
6
LBNL
Berkeley, California
2005
7
UCLA
Los Angeles, California
2006
8
UCSB
Santa Barbara, California
2007
9
UCB
Berkeley, California
2007
10
UCR
Riverside, California
2008
1.4 The WMKO Remote Observing Model WMKO adopted its existing remote observing model4 in 1996, and for the past decade it has provided the foundation for all of the Observatory’s remote observing efforts. The model is based on two fundamental principles: 1. All instrument control applications (including user interfaces) run on summit control computers. 2. The displays generated by those applications are re-directed to the remote observing sites (See Fig. 3). Regardless of whether they are observing from the Mauna Kea summit, from Waimea, or from a remote observing facility on the mainland, observers are accustomed to operating instruments at WMKO while seated at a local workstation equipped with at least three monitors (see Figs. 1 and 2). Typically, these monitors are allocated to displaying the user interfaces and image displays associated with the selected instrument, and a fourth monitor (often attached to a second workstation) is used to display the real-time images from the guide camera, plus the status and meteorological displays provided by the Observatory’s telescope and dome control systems. Also regardless of the location from which they are working, observers are prohibited from remotely operating the instrument until WMKO staff at the summit have finished configuring it for that night’s observing and have officially released it for use. The current release status for the instrument on each of the two telescopes is posted on WMKO’s Summit Instrument Activities Schedule (SIAS) web page which is accessible to the remote observers.
http://www2.keck.hawaii.edu/inst/siastng/release/web/Sias/siasEh.php
2. THE KVNC SOFTWARE SUITE Allowing observers at disparate sites to control a shared instrument requires some mechanism for allowing multiple users simultaneous access to the instrument control interface. One way to accomplish this would be to provide each site with its own independent instance of each of the numerous widgets and applications needed for instrument control. As described in x4 of our 2006 paper,6 we chose a different approach: running a single instrument interface inside a sharable desktop environment called VNCy . This approach not only allows observers at multiple sites to control the instrument, but also allows each to see the others’ actions in real time. In practice we find that the sharable viewers allow observing teams to interact more effectively, permit students participating remotely to get a better feel for operating the instrument, and enable our Hawaii-based support astronomers to better assist remote observers and troubleshoot observing problems. Although the VNC distribution performs its task well, we desired additional functions to simplify the process of starting and stopping the numerous servers and viewers associated with the instrument interface. To maintain flexibility, we wanted to avoid hardwiring the software to run VNC servers on specific ports; instead, we allow the software to select the next available port when starting a server. This dynamic selection of servers presents a bookkeeping problem which motivated us to write a collection of custom scripts, which we call the KVNC suite, to manage the operation of the servers and viewers.
2.1 The KVNC software at WMKO The KVNC suite includes the following noteworthy scripts: kvnc is the top-level script which starts and stops the full complement of VNC desktops and viewers for the current instrument. Typically this is the only command the user invokes to manipulate the VNC windows, using a directive such as kvnc start servers. kvncserver starts or stops a VNC server process of the requested type (pseudocolor or truecolor) and assigns the server an appropriate name, thus permitting other applications to refer to it using an alias instead of a display number. kvncviewer starts a VNC viewer process connected to the requested server, which is identified by its alias. kvncinit configures the current account to use the KVNC suite by copying appropriate files from a central repository. It’s typically run by the same automated script we use to clean out and prepare observer accounts for use by the next observing team. kvncstatus lists the currently-running VNC viewer and server processes. The KVNC scripts are driven by configuration files, thus enabling the same software suite to serve the entire complement of observatory instruments without any need for instrument-specific conditional code in the software.
2.2 The KVNC derivatives at the mainland sites Additional scripts facilitate the starting and stopping of VNC viewer windows at mainland sites. The functions of these scripts are to communicate with the summit-based machine running the VNC servers, initiate ssh tunnels to allow the VNC viewers to be displayed securely on the remote hosts, and launch the appropriate complement of viewers on the remote machine. The scripts in the mainland KVNC suite include: kvncinst to launch the VNC viewers displaying the instrument control interface. kvnctel to launch the VNC viewer displaying guider, telescope, and meteorological status. kvncall to launch both the instrument and telescope VNC viewers . This script is used only at those sites which offer a single 4-headed display instead of two separate machines for telescope and instrument windows. y Virtual Network Computing ( VNC 7 ) is a thin-client system in which a generic VNC client (the VNC viewer, with which the user interacts) offers remote access to a sharable virtual desktop provided by a VNC server; multiple VNC viewers can connect to a single, shared VNC server, and all such viewers provide comparable views of and access to that desktop. VNC servers can offer either 8-bit of 24-bit X visuals, but unlike modern X servers, cannot offer both simultaneously.
3. SCHEDULING While software and hardware are obvious components of the remote observing setup, communication between human beings at WMKO and the remote sites is an easily overlooked — but nevertheless essential — element in the successful operation of the system. Careful coordination and cooperation helps prevent scheduling conflicts in the planned usage of the remote facilities and ensures that all necessary hardware and software is fully functional prior to the observing run.
3.1 The role of the WMKO Mainland Observing Coordinator The Mainland Observing Coordinator at WMKO is the linchpin in the process of handling requests for use of the mainland observing sites. Observers wishing to make use of a mainland site submit a request to the coordinator, who acknowledges receipt of the request and then consults with both the WMKO support astronomer assigned to that observing team as well as the manager of the selected mainland site to determine whether the request can and should be accommodated. Although the vast majority of requests present no problem, a request may be denied due to unavailability of the requested mainland site, insufficient experience with the specified instrument by the proposed observing team, or inadequate lead time to arrange for a pre-observing-run test at that remote site. Once the mainland site manager and support astronomer approve the request, the coordinator sends email to the observer notifying him or her of the approval and arranging a date and time for a “dry run” at which the observers should be at the mainland site to help test the mainland observing setup prior to their run. The coordinator must also notify the mainland site manager of the approval and of the date and time of the proposed test, and must ask the maintainer of the WMKO telescope schedule to note on the schedule that the team will use the remote site for observing. The coordinator is also responsible for maintaining and monitoring the results produced by WMKO’s mainland status script that is invoked on a host in Waimea each morning via the Unix cron facility. That script uses the Unix ping utility to verify that all of the relevant ISDN routers and remote observing workstations at each of the mainland remote observing sites are reachable from Waimea via the network. If any are not, the script sends an email message to the coordinator and to the manager at the relevant site to alert them of a potential problem. Since some remote observing hosts often go unused for many days at a time, such automated checking helps to provide timely detection of problems that might otherwise go unnoticed until a remote observing session is just about to commence.
3.2 The role of the managers at each mainland site The mainland site managers play a critical role in the process. First, they must ensure that the computers and videoconferencing equipment at the mainland site remain in good working order. Second, they must track the scheduled usage of their site to ensure that no conflicts arise between observers intending to carry out observations at WMKO and those wishing to use the same facility for observing at other facilities; for example, many of our UC sites also are used for remote observing with the telescopes at the Mt. Hamilton station of UCO/Lick Observatory. Third, they must be available to instruct new observers in the process of starting up and shutting down the VNC windows using the scripts described above for mainland operation. Finally, they must be available during the night to troubleshoot problems with the equipment and local area network at the mainland site. Most site managers have other responsibilities and spend only a small fraction of their time performing these functions. Also, at some sites, the manager’s responsibilities are split between two individuals, with one providing technical support and the other handling local administration and scheduling. The remarkably low frequency of problems at the remote sites speaks to the dedication of the mainland site managers as well as the reliability of the system design and the deployed equipment.
3.3 The WMKO mainland remote observing schedule The WMKO Mainland Observing Coordinator maintains a calendar of upcoming remote observing in order to prevent potential conflicts in usage, such as multiple observing teams wishing to use the same mainland observing site to observe with different telescopes on the same nightz . This schedule is posted on the WMKO websitex , separately from the main telescope schedule, to allow observers and mainland site managers to track usage of the system. z At one time it was necessary to prevent scheduling “mainland only” mode (see x5) observing on both telescopes at the same time; however, this restriction has been eliminated now that WMKO offers two full sets of ISDN backup lines and can thus support mainlandonly mode observing simultaneously on both telescopes. x http://www2.keck.hawaii.edu/inst/mainland observing/mainland observing schedule.html
3.4 Integration with the main WMKO telescope schedule In addition to being tracked on the Mainland Observing schedule, use of the mainland facilities is noted on the main WMKO telescope schedule{. This helps remind the support astronomer, observing assistant, and other WMKO staff that the observers will be operating the instrument from the mainland. It also alerts WMKO administrative staff that they need not contact such observers about reserving dormitory rooms in Waimea.
3.5 Split nights Nights on the Keck telescopes are considered so valuable that observers are often allocated fractional nights in order to give more astronomers access to the telescopes. Shared nights on the telescope present a special challenge when one or both teams wish to use a mainland site. Our experience has been that once the VNC sessions are running they are highly reliable, and that problems most often occur at startup; hence, we strongly discourage teams from starting or stopping VNC sessions during the night. If only one team wishes to operate from the mainland then we typically require the other team(s) to also operate the instrument software in the shared environment. Given the recent increase in bandwidth between the Mauna Kea summit and Waimea, system performance is now nearly as good within VNC as outside of it, and hence observers in Waimea are unlikely to notice any significant degradation in the response of the software when running under VNC.
3.6 Split instruments Certain observing teams prefer to switch among the multiple available infrared instruments on Keck II during the night in order to make optimal scientific use of their observing time. When these teams include observers on the mainland, it becomes necessary to run VNC sessions for more than one instrument at the same time. Although this capability was not explicitly designed into the system at the outset, the scripts we’ve written were easily able to accommodate this observing mode. Observing teams can easily launch additional VNC viewers for each instrument they plan to use and simply minimize the inactive viewers when not in use.
3.7 Multi-site observing teams Given the increasingly collaborative and multidisciplinary nature of observing teams, some programs involve astronomers at multiple sites on the mainland, all of whom want to participate in the observing. Again, supporting this mode of operation was not explicitly designed into the system, but the flexibility of our software easily permits this. The principle of operation is identical whether two or more sites are involved, and hence no different action is required to start and stop the VNC sessions.k However, performance of the system may degrade as more mainland sites view the same VNC servers, and the interface may become noticeably sluggish at all sites if a single site with poor bandwidth enters the mix. Fortunately, with the recent improvements in bandwidth from Mauna Kea to the mainland, such potential slowdowns have not been observed in practice.
4. USER TRAINING Prior to the advent of the KVNC software suite, observers could only access the full instrument control interface while in Hawaii. With VNC now providing the capability to display interfaces back to the mainland, observers unfamiliar with a given instrument can obtain experience in using it without traveling overseas. In the days preceding a mainland observing session, we typically arrange a pre-run checkout of the instrument interface from the mainland site which serves to verify that the instrument control software, KVNC scripts, and videoconferencing units at both ends are functioning well. Since observing support for remote observers is provided by WMKO astronomers rather than by the staff of the mainland site, the pre-run checkout presents a valuable opportunity to introduce observers to the instrument control interface while the support astronomer is on the video link. Increasingly, students and observers new to an instrument are choosing to eavesdrop on observing runs being carried out in Hawaii by more experienced colleagues. For a novice to be able to watch “over the shoulder” while experts observe can be a very valuable way to learn how to use the instrument without having to stay up all night or travel long distances. { http://www2.keck.hawaii.edu/schedule/index.php
k The number of mainland sites that can effectively operate together as a distributed multi-site team may be limited by the multi-
conference capabilities of the video-conference hardware at each of the sites. In all cases, the current hardware will support multi-site teams involving two mainland sites, and teams involving more sites could be accommodated by upgrading such hardware as needed.
5. USAGE METRICS AND STATISTICS When the mainland remote observing facilities were first conceived, two distinct modes of operation were envisioned and subsequently articulated in WMKO’s formal policy: 1. Remote eavesdropping mode
At least one member of the observing team observes from Waimea, while the other members of the team observe from the mainland. Observer(s) in Waimea have primary responsibility for instrument operation, but observers on the mainland are able to operate the instrument if desired. 2. Mainland-only mode
All members of the observing team observe from California site(s). California observer(s) have sole responsibility for instrument operation.
Figure 4. Monthly usage of the WMKO mainland observing system between June 2005 and June 2008. Here, and in Figs. 5 and 6, the bottom part of histogram represents eavesdropping mode usage and the top part represents mainland-only mode operation. Since early 2007, system usage has been steadily growing, with mainland-only mode representing an increasing fraction of remote observing usage.
In reviewing the use of the mainland remote observing facilities over the last three years (see Fig. 4yy ), several trends become apparent. First, the majority of observing teams that have utilized the mainland facilities have employed the first mode of operation. While the mainland-only mode has been used from six of the eight mainland sites (LBNL, UCSC, UCSD, UCLA, UCB, and CIT), before 2007 such usage accounted for less than 10% of mainland remote observing activity. http://www2.keck.hawaii.edu/inst/common/mainland observing.html yy The number of “nights” displayed along the y axis is computed as the sum of telescope-night-fractions summed across both Keck
Telescopes and across the number of nights in the given month. The telescope-night-fraction value for a given night on a given telescope represents the fraction of that night (e.g., 0, 0.5, or 1.0) on which the observing program for that telescope included participation from observers working from at least one of the eight mainland remote observing sites.
Figure 5. Usage of the mainland observing system by site since February 2006.
Figure 6. Usage of the mainland observing system by instrument since February 2006.
Second, as expected, the number of nights per month that involve WMKO remote observing from the mainland has grown significantly as more mainland sites have come online. For example, for 2006, remote observing from mainland sites was conducted an average of 9.6 nights per month, while for 2007 this figure rose to 12.6; the first six months of 2008 have averaged 19.7 nights per monthzz . Third, an increasing fraction of those nights involved operations conducted from two (or more) different mainland sites on the same night. Such operations have come about in several different ways. Since there are two Keck Telescopes, two distinct observing programs can take place each night, and each such program could involve observer(s) located at a distinct mainland site. Alternatively, a single observing program involving only one instrument and telescope may involve a multi-institution observing team in which team members participate from more than one mainland site. In addition, for a given instrument on a given telescope, some nights are scheduled as “split nights,” where the observing time is split between two (or more) different programs, each with their own observing team; in such cases, each of the two remote sites would operate the instrument for only a portion of the night.
6. FUTURE PLANS As described below, we envision several future changes to improve the operation, maintainability, performance, and reliability of the WMKO remote observing system.
6.1 Future plans and directions at WMKO 6.1.1 Dedicated VNC Servers We are now in the process of installing at both Keck I and Keck II separate server hosts that will be dedicated solely to running VNC servers to support remote observing. As an interim measure, we have been running VNC servers directly on some of the newer instrument control computers, but overall administration and access control issues will be simplified once the VNC servers can be relocated to dedicated hosts at each telescope. This may also improve VNC performance and reduce the risk that VNC software will interfere with operating the instrument control computers. 6.1.2 GUI-based KVNC Although the script-driven KVNC suite is reliable, the process of starting and stopping the VNC sessions it is not yet as convenient as it should be for the observers and support staff in Hawaii. Furthermore, observers will sometimes want to deviate from the “normal” mode of operation by starting only certain VNC viewers, or by launching the viewers in “passive” (read-only) mode so that they can watch the observations taking place without any risk of issuing unintended commands, i.e., to “look but not touch.” To make these options easier to select, we plan to provide a simple interface written in Tcl/Tk for launching the KVNC scripts. zz Month to month fluctuations reflect a variety of factors, such as telescope time assigned to observers affiliated with institutions lacking their own remote observing facility or nights allocated to interferometry, segment phasing, or telescope engineering.
6.1.3 Making VNC the default As we gain more experience running the instrument software in a sharable mode, we are coming to appreciate the benefits of having the control interface always running and accessible from anywhere, not least of which is the ability to troubleshoot instrument problems more efficiently when on call at night from home. In fact, one of our instruments (OSIRIS) is now operated solely in VNC mode. Given the positive experience within the observatory in operating instruments in this manner, it is likely that we will eventually run all of our instruments under VNC at all times, thus eliminating the need for observers in Hawaii to perform special start-up procedures on mainland observing nights. Existing WMKO policy assigns to each observing team its own account to use for operating an instrument during a given observing run; a move to support full-time operation of instruments under VNC may require changing the manner in which those accounts are used. This model was intended to help segregate observing data generated by different observing teams, and to keep pre-observing and post-observing teams from interfering with each other as well as interfering with the preparations, calibrations, and observations being carried out by that night’s observing team. Given that the typical instrument control interface requires 8 VNC servers, keeping VNC servers running for numerous accounts on each instrument at all times would be unwieldy. Instead, we are considering designating one account for operating each instrument, thus reducing the required number of servers to a manageable number. This model of account usage has been employed with NIRSPEC since its inception and has worked well. Observing teams would continue to use their designated account for pre-observing and post-observing activities. 6.1.4 Greater involvement by observing assistants Although WMKO’s observing assistants (OAs) are well versed in the operation of our telescopes, they remain largely unfamiliar with instrument operations because — being based at the Mauna Kea summit rather than in Waimea with the observing teams — they may never get to see the instrument control interface in operation. This is one unfortunate consequence of the switch from summit observing to Waimea observing that occurred over 10 years ago. However, VNC viewers can allow the OAs to eavesdrop as the observers operate the instrument, thus teaching them how to run our instruments and, hopefully, improving their ability to form a first line of defense in instrument troubleshooting during the night. 6.1.5 Automated request tracking The process of requesting use of the mainland observing sites and obtaining approvals from support personnel is currently handled via email. The growing popularity of the system reported above presents an increasing administrative burden on the Observatory staff. A system now under design will automate the request/approval process through the use of web forms and database storage, freeing staff time now spent tracking requests.
6.2 Future plans and directions at mainland sites 6.2.1 Additional sites Additional mainland remote observing facilities are now being implemented at two other campuses of the University of California system: Davis and Irvine. In addition, such facilities may also be implemented at the sites of other institutions affiliated with WMKO (e.g., the Institute for Astronomy at the University of Hawaii) if there is sufficient interest from the community of observers at those locations. WMKO is fortunate to draw interest from astronomers all over the world, some of whom have expressed interest in observing from their home institutions rather than traveling to Waimea or to one of our established mainland remote observing facilities. We currently deny such requests because the effort required to install and test the software at a new site is significant. Also, unlike our existing mainland facilities, other sites do not offer the ISDN backup circuits which maintain network connectivity during Internet outages. It is likely that we will continue requiring our observers to use the established mainland sites for remote observing rather than permit observing from random locations. 6.2.2 Interaction with other observatories Seven of the eight existing Keck mainland remote observing facilities are affiliated with the University of California. Many of the observers at those sites also observe with the Lick Observatory telescopes located on Mt. Hamilton, California. On nights when those facilities are not assigned for use with the Keck Telescopes, they are available for use with the Lick telescopes. Such dual use is possible because the Lick Observatory remote observing system is modeled on the one developed for Keck.8 If scheduling conflicts occur, Keck-related usage takes precedence.
6.2.3 Multiple remote observing rooms at some sites Unlike Waimea, which has two separate remote observing rooms (one for each of the two Keck Telescopes), each mainland facility (with the exception of UCB) is currently equipped to support only a single telescope/instrument at any given time. On most nights this is not a problem because the observing teams using the Keck I Telescope and those using Keck II are not associated with the same mainland site. However, on a small fraction of the nights, the observing teams on both telescopes are from the same mainland site. If such conflicts become more frequent, some sites may follow the lead of UCB and establish a second remote observing room. At UC sites, a second room would also reduce potential scheduling conflicts between Keck- and Lick-related usage. 6.2.4 Improving software configurability The current process for maintaining and updating the scripts used at the remote sites is somewhat cumbersome and inefficient. Since each mainland site is independently administered, computer configurations differ from location to location. In the near future we hope to adapt our software to employ site-specific configuration files instead of hand-modifying the scripts for different sites. We are also investigating possible mechanisms for automating the process of distributing new versions of the various scripts to the mainland sites. 6.2.5 Streamlining pre-run checkout Since the inception of this system, we have generally performed a full pre-run checkout of the VNC system from the mainland site prior to the start of the run. Although typically pro forma, these checkouts have occasionally identified problems which would have interfered with observing; thus, we have felt compelled to continue doing them. As we gain increasing confidence in our system, we hope to take a more judicious approach to decide when checkouts are required. Regular observers using common instrument/site combinations will not require this step, whereas observers unfamiliar with the system or those using instrument/site combinations which have not recently been used will be flagged for pre-run checkout.
7. REFLECTING ON REMOTE OBSERVING To a great extent, the deployment of a remote observing capability at WMKO has delivered on its initial promise. Not only are astronomers now freed from observing in the hypoxia-inducing conditions at the Mauna Kea summit, they are also liberated from enduring the often-dismal experience of modern air travel in order to acquire their observations. Even some astronomers who never planned to use the mainland remote observing facilities have been pressed into observing there due to causes as diverse as medical emergencies, terrorist attacks , and the sudden assignment of observing time due to reallocation of WMKO engineering nights. In these more environmentally-aware times, any technology that reduces the “carbon footprint” of our astronomical activities is welcomed, and the WMKO remote observing system certainly does so. However, remote observing is not necessarily an unmitigated blessing. Others have already pointed out some of the potential pitfalls of remote observing,9 and we freely confess that it was with some trepidation that the WMKO staff agreed to support this grand experiment several years ago. In addition to the extra effort required by our staff to manage remote observing sessions, a major concern for our small, isolated band of support astronomers was that the observing teams might cease traveling to Hawaii, thus depriving us of meaningful professional interaction with our colleagues. Fortunately, this has not (yet) come to pass: until now, observing teams have mostly used the remote observing system to involve additional colleagues in the observing process while still sending members to Hawaii for their runs. The Visiting Scientist Quarters in Waimea remain as full as ever these days (thanks in part to the increasing number of split nights that double the numbers of observing teams on a telescope). This situation may not continue indefinitely. Until now, many observers have regarded the remote observing capability as still somewhat experimental and have thus continued sending team members to Hawaii in case of potential problems with the setup. The growing confidence of observers in our remote observing system, coupled with the substantial recent increases in the cost of air travel to Hawaii, could result in a sudden and significant reduction in the number of astronomers who travel to Hawaii for observing runs at WMKO. In fact, at this writing we note that a record number of requests for mainland observing have been received for next month (June 2008). Whether this truly represents a trend in response to economic circumstances or is merely a statistical anomaly remains to be seen. As much as we enjoy seeing astronomers use
See http://currents.ucsc.edu/01-02/09-17/crisis keck.html
the remote observing system we have worked so hard to design and implement, we hope that the promise of a few precious days away from the office to concentrate on science, the allure of uninterrupted daytime sleep in comfortable quarters, and the attraction of a visit to a sandy beach will continue to entice most astronomers to make the journey to Hawaii to use the Keck telescopes.
8. APPLICABILITY TO OTHER OBSERVATORIES The solutions we have developed to support mainland observing with the Keck telescopes are in principle easily adaptable to other facilities with similar instrument control interface designs, such as the planned Thirty Meter Telescope (TMT), and, as noted in the previous section, have been adapted for use at Lick Observatory. Given the likelihood that observing runs on the next generation of large telescopes will be significantly shorter than those currently scheduled at WMKO, and assuming that fossil fuel prices continue to increase the cost of trans-oceanic travel, it is almost certain that observers will want to operate these new telescopes from remote sites starting at first light (and prior to that, engineers will debug instruments remotely during the commissioning phase). If the new generation of observatories is to make best use of their remote observing capability, the design work required to support this mode should be considered early in the project.
9. CONCLUSION The original goal of this project was to enable mainland-based observers to operate the instruments on the Keck telescopes reliably and conveniently in a manner as similar as possible to observing from Waimea. We believe we have generally accomplished this objective. The steadily increasing demand for and near-perfect reliability of our mainland observing system is testament to its success. However, the ability to share the instrument control windows among people at different locations has important additional benefits to the WMKO itself outside of observing: improved ability to share information between support astronomers in Waimea and technical staff at the summit; better interaction with instrument development teams on the mainland; greater ability for on-call support astronomers to perform nighttime instrument troubleshooting from home; and the potential for higher instrument awareness for our observing assistants. While several aspects merit improvement as described above, the system overall is performing well. Ongoing operation of the remote observing system comes with a certain cost to the Observatory, but our plans to modify the system will further reduce the staff burden required to track usage requests, perform pre-run checkouts, and manage remote observing sessions. In retrospect, we feel that the design of our remote observing system is fundamentally sound, offers significant benefits to both the observers and the Observatory at very modest cost, and could easily serve as the basis for similar projects at other existing and planned observatories.
ACKNOWLEDGMENTS We wish to acknowledge the combined and ongoing efforts of the University of Hawaii, the Gemini Telescope consortium, the National Science Foundation, the University Corporation for Advanced Internet Development, and the Corporation for Education Network Initiatives in California (CENIC) toward the establishment and maintenance of a reliable, highbandwidth network pathway between the Mauna Kea summit and the U.S. mainland. We also thank the following WMKO staff who contribute to the ongoing operation of the remote observing facilities in Waimea and in California: support astronomers Randy Campbell, Al Conrad, Grant Hill, Marc Kassis, and Hien Tran; software engineers Liz Chock and Julia Simmons; and system administration manager Jon Chock. We also acknowledge the efforts of the following individuals for their efforts in setting up the various remote observing sites in California. Caltech: Judy Cohen and Patrick Shopbell; LBNL: Deb Agarwal, Saul Perlmutter, and Tony Spadafora; U.C. Berkeley: Andrew Howard, Kelley McDonald, and Geoff Marcy; U.C. Los Angeles: Matthew Barczys, Chris Johnson, Leon Kopelevich, Ian McClean, and Jason Weiss; U.C. Riverside: Bill Strossman, Gillian Wilson, and Gabriela Canalizo; U.C. San Diego: Peter James, David Kirkman, John O’Meara, and David Tytler; U.C. Santa Barbara: Glenn Schiferl, Tommaso Treu, and Crystal Martin; U.C. Santa Cruz: Marc Boolootian, Kyle Lanclos, Jeff Mallory, and Jim Warner. The W. M. Keck Observatory is operated as a scientific partnership among the California Institute of Technology, the University of California, and the National Aeronautics and Space Administration. The Observatory was made possible by the generous financial support of the W. M. Keck Foundation.
REFERENCES [1] Kibrick, R. I., Allen, S. L., and Conrad, A. R., “Remote observing with the Keck Telescopes from the U.S mainland,” in [Advanced Global Communications Technologies for Astronomy], Kibrick, R. and Wallander, A., eds., Proc. SPIE 4011, 104–116 (2000). [2] Kibrick, R. I., Hayes, B., Allen, S. L., and Conrad, A., “Remote observing with the Keck Telescopes from multiple sites in California,” in [Advanced Global Communications Technologies for Astronomy II], Kibrick, R. I., ed., Proc. SPIE 4845, 80–93 (2002). [3] Kibrick, R., Conrad, A., and Perala, A., “Through the Far Looking Glass: Collaborative Remote observing With the W. M. Keck Observatory,” ACM Interactions 5(3), 32–39 (May-June 1998). [4] Conrad, A. R., Kibrick, R., and Gathright, J., “Remote Observing with the Keck Telescopes,” in [Telescope Control Systems II], Lewis, H., ed., Proc. SPIE 3112, 99–110 (1997). [5] Cohen, J., Shopbell, P., and Bergman, L., “Remote Observing with the Keck Telescope Using the ACTS Satellite,” in [Final Report for NASA Grant BK-509-20-42-00-00 ], 1–22 (March 1, 1998). [6] Kibrick, R. I., Allen, S. L., Conrad, A. R., and Wirth, G. D., “Optimizing interactive performance for long-distance remote observing,” in [Advanced Software and Control for Astronomy], Lewis, H. and Bridger, A., eds., Proc. SPIE 6274 (2006). [7] Richardson, T., Stafford-Fraser, Q., Wood, K. R., and Hopper, A., “Virtual Network Computing,” IEEE Internet Computing 2(1), 33–38 (January/February 1998). [8] Grigsby, B. J., Chloros, K., Gates, J., Deich, W. T., Gates, E. L., and Kibrick, R. I., “Remote observing with the Nickel Telescope at Lick Observatory,” in [Observatory Operations: Strategies, Processes, and Systems], Brissenden, R. J. and Silva, D. R., eds., Proc. SPIE 7016-79 (2008). [9] Lockman, F. J., “Can Remote Observing be Good Observing? Reflections on Procrustes and Antaeus,” in [Observing at a Distance], 325, World Scientific (1993).
