Ben Penaflor is computer systems manager at General Atomics, which operates the DIII-D National Fusion Facility in San Diego, California, for the US Department of Energy. This post is part of a series on how the COVID-19 pandemic is affecting the personal and professional lives of physicists around the world. If you’d like to share your own perspective, please contact us at pwld@ioppublishing.org.
I’m part of a team that provides support for computer, control and data acquisition systems for plasma operations at the DIII-D tokamak. Before the pandemic, I would arrive at the fusion research facility at around 7 each morning to check the readiness of our systems before attending a standing-room-only meeting. Other attendees included physicists overseeing and participating in the experiment, plus representatives from operations support groups in power, vacuum, heating, water, cryogenics, diagnostics and computer systems – as many as 60 people in total.
After that, I would spend most of my day in the DIII-D control room. This was a busy environment, with every seat taken and all eyes focused on large display monitors showing live tokamak status and experiment results. The physics session leader overseeing the experiment would sit at the front, working closely with other physicists and with the chief operator who oversees tokamak operations and safety. At nearby stations, other groups of scientists would monitor diagnostics and analyze results. These individuals would be in close contact with the session leader, often having direct personal conversations to provide immediate assistance and feedback.
In addition to the staff in the main control room, we had teams spread throughout the facility to support various aspects of the tokamak’s operation. These groups kept in contact via a direct overhead paging system, landline phones and walkie talkies. Doing experiments on DIII-D requires input from a lot of people, so there could be as many as 120 of us in the facility during a day of experiments.
From hands-on to mostly remote
The pandemic changed all of this. Since the start of restrictions, most fusion research staff have been working offsite, with only a few allowed in the DIII-D facility. We were, however, fortunate in that DIII-D staff had extensive experience in working with other tokamaks across the world and even, in at least one case, controlling a tokamak remotely from San Diego. This knowledge gave us a good base on which to build.
The first step was to provide offsite staff with access to the full set of computer systems and applications needed to run experiments in a secure manner. Since quite a few of these tools control critical pieces of hardware, it was crucial that access was restricted to authorized personnel.
The next step was to create a virtual control room that would allow offsite staff to see and hear the same information they were accustomed to having at the facility. We did this by adding to and improving upon an existing set of web-based displays showing live tokamak status information, real-time signal plots, and a tokamak plasma cross-section animation. We also set up two live camera feeds of the DIII-D control room along with broadcasts of the on-site paging system.
The final and most crucial step in setting up remote operations was to provide a communications infrastructure that could accommodate coordination amongst a large and varied group of individuals working both onsite and offsite. Initially, we tried having everyone in a single Zoom meeting room, but this quickly ran into problems. Communications during experimental operations tend to be very ad hoc and dynamic, and as the number of people in the meeting grew, it became very difficult to manage these kinds of conversations. With close to 100 people signing on, it was hard for individuals to break into general discussions or for smaller groups to have conversations amongst themselves – the friction involved in jumping from conversation to conversation was just too high.
A new use for a gaming app
Instead, we settled on Discord, an application that is mainly used in the video-gaming community. Our Discord server provides a platform for DIII-D group members to communicate on an as-needed basis, as well as listen in and participate in the main experiment. Discord’s architecture makes it possible for many users to “meet” in one place, while also allowing them to break off into smaller groups and move between separate dedicated channels. It proved very effective in bringing all the different DIII-D groups together.
Another problem that Discord helped solve related to contacting users working offsite. Thanks to Discord, people who could no longer answer phones in their offices could be reached easily by voice or text while signed into the app. While the initial adaptation and deployment was a challenge – we had a large number of users requiring access to Discord, and they all needed to be registered and trained in order to fit it into their day-to-day methods of conducting business – it has been hugely beneficial to our operations.
Telecommuting to a tokamak
Thanks to our remote-operations solutions, most of our scientific and support staff can, in effect, run the tokamak without having to step outside their homes. Scientists can analyze all aspects of the plasmas and status of the tokamak in real time, while coordinating and collaborating with one another using tools such as Discord and Zoom.
Of course, there are still some activities that require onsite staff to touch hardware in the lab. These include calibrating equipment at the start of each day, performing routine maintenance, and making necessary repairs and upgrades. To limit the number of people engaged in these tasks, we provided onsite workers with tablets equipped with cameras and microphones to connect them with remote experts who could offer guidance in repairs or adjustments needed to hardware.
What’s missing – and what isn’t
Although the face-to-face camaraderie of working alongside colleagues in our quest for fusion has been much missed, we’ve found ways to adjust. We’ve also been a bit surprised at how easy and natural it’s become to do high-level physics experiments from our homes. In some cases, the transition to remote operations has even made our work more efficient. One example is the improved communications between different groups. It is now much more straightforward to find and contact the right people to resolve an issue thanks to our new Discord-based central communication system.
Physics in the pandemic: making particles from space tangible for schoolkids
It remains to be seen whether some people will grow less inclined to sign on to communications tools such as Discord as more staff become available onsite. However, the improvement in overall communications efficiency it offers should give people an incentive to continue using it.
More generally, it is easy to see how many of the tools developed to support remote operations during the pandemic could be left in place as more staff return to onsite working. The live web-based status displays, real-time plots and control room videos are as useful and effective to users working from home as those working from their own offices. These tools give us greater flexibility and efficiency in supporting operations and monitoring the experiment.
