TVA's new deal: From call center to world-class ops center

The Tennessee Valley Authority, a government corporation that provides electricity to 155 distributors for more than 9 million people in seven states, upgraded its old operations hub to a full-fledged IT Operations Center with the September opening of a 12,000 square-foot, purpose-built facility in Chattanooga.

“It was more of a call center than anything else,” senior IT customer and infrastructure operations manager Keith Youngblood said of the old facility.

The new center is a co-located service and operations center that triples the size of the previous facility. It supports not only the front-line IT service providers, but also IT event and incident management teams, as well as change management teams. It allows near real-time monitoring and surveillance of the TVA’s wide-area and local-area networks and includes an emergency operations center that can be activated for natural or other disasters that threaten or affect the electric grid for the eastern half of the nation.

The effort required moving the support team into cramped quarters during the build-out of the new facilities, but the improvements justified the inconvenience, said project manager Jim Bingel. “They were open to squeezing into a temporary facility for nine months,” Bingel said.

The TVA is a first-of-its-kind government corporation, created in 1933 as part of President Franklin D. Roosevelt’s New Deal to manage natural resources and improve economic conditions in the Tennessee River Valley. As part of the project it built dams and generated electricity, and today is the country’s largest public power company, servicing most of Tennessee and parts of Alabama, Georgia, Kentucky, Mississippi, North Carolina and Virginia. It operates 11 coal-fired generating plants, nine combustion turbine plants, three nuclear plants and 29 hydroelectric dams.

With main offices in Knoxville, Chattanooga and Nashville, and Muscle Shoals, Ala., it also is the controlling entity for the eastern half of the U.S. power grid, forming a critical part of the national infrastructure.

Although it is a government corporation (its main website http://www.tva.gov/ is in the .gov domain), it is self-funding today and receives no tax money. Operating revenue from electricity sales totaled $11.7 billion in 2011.

 “It met the needs at that time,” Youngblood said of the previous TVA operations center, which dated to the early 1990s. It was primarily a help desk for the system’s network, which serves about 2,500 endpoints and two data centers, in Knoxville and Chattanooga. Some problem resolution was done on-site, but mostly it served as a dispatch service for field technicians who did the actual trouble shooting and resolution.

“We improved over time,” he said, but with the emergence of new threats and risks, the network needed near real-time monitoring and surveillance to detect and respond to disruptions caused by weather as well as attacks and security breaches. “The center did not give us the space for the technology we needed for the world-class center we envisioned.”

In 2010 the authority began considering location options for the new facility and concluded that it should remain in the Chattanooga complex, Bingel said. Six months of construction began in 2011 and the existing center was squeezed into space on another floor while its old home was remodeled. The space was gutted and expanded from one sixth of the floor to half, and new technology was installed.

The new facility supports 80 staff positions and five management and administrative positions on the main floor, and has a large conference room that serves as an emergency operations center when needed, as well as two smaller conference rooms. Each workstation has three monitors and its own uninterruptable power source, and the conference rooms have large-screen monitors.

The center is dominated by a video wall with a three-by-six matrix of high-definition flat screen monitors from Planar Systems for monitoring news and weather feeds for conditions that could affect the power grid in the Southeast. “That’s critical for our people to know in advance,” said Erik McGann, manager of IT operations.

The entrance area of the room is occupied by the front-line call receivers. Behind them is the problem resolution center, which gets the more difficult problems. The event and incident management teams in the back control the video feeds.

Sight and sound were primary considerations in designing the center, Youngblood said. Furniture and work stations were kept low to ensure clear lines of sight to the video wall. And with 50 to 60 people in an open space, many of them handling incoming telephone calls, noise also is an issue. The center is designed with three audio zones to control and target sound input.

Key lessons learned in building out the operations center were to plan and build for the organization’s mission and also for the future, Youngblood said. The goal was to create a facility that would not be outgrown in 20 years. “Hopefully we built the center so it can be modified for future needs,” he said.