In shared environment, USAF builds secure, stable IT flight deck

A couple of years ago, the IT environment of the Air Force National Capital Region was becoming volatile and unstable.

“We were doing day-to-day firefighting” to keep the systems up and running, and many of the servers were nearing their end-of-life,” said John Rogers, deputy commander of the Air Force’s 844th Communications Group. “We were at a crossroads,” he said. The group had to either buy more servers or find a more efficient solution.

Choosing the best path depended on addressing head-on the limitations of its current infrastructure: Servers are expensive to buy and to operate. Each is assigned to a single user and typically operates at 10 to 15 percent capacity. But excess capacity cannot be reassigned or easily shared, resulting in wasted resources. Configuration and maintenance often are also poorly documented, complicating administration and security.

The 844th set out to create a stable, secure environment that was simplified, scalable and in line with the Defense Department’s Joint Information Environment, a construct comprising  a single security architecture with a shared IT infrastructure and enterprise services. The result is the AFNCR Shared Computing Environment (SCE). “I don’t call it a cloud,” Rogers said. “I call it a shared environment where I provide platform as a service.”

Started in late 2012 with Lockheed Martin as the prime contractor, the SCE now is winding up the second of four deployment phases. So far, it has delivered the cost efficiency hoped for and some of the improvement in quality of service. In coming phases, the Air Force hopes to improve business agility as the environment moves to software defined networking and storage.

The AFNCR’s IT department supports 20,000 customers at 25 locations in the Washington, D.C., area. To serve them, the SCE has mirrored sites at Andrews and Bolling Air Force bases, using EMC VPLEX and VMware Storage vMotion to enable movement of data from virtual machine to virtual machine and across arrays and sites.

Both NIPRNet (Nonsecure IP Router Network) and SIPRNet (Secure IP Router Network) are housed in each of the sites, which Rogers said was something of a feat.

“There are some challenges for how you secure it,” he said of the classified SIPRNet servers. But the shared environment has the capability of providing greater security because there no longer is a client hard drive to secure. A virtual desktop accesses data on a secure server.

In addition to Lockheed Martin, which received a contract to provide communications, information services and user support in the AFNCR, partners in the SCE program include Cisco, Microsoft and VMware.

The first phase of the SCE program, to build and consolidate, began in October 2012 and was completed the following June. The Air Force shut down 400 end-of-life servers and replaced them with a shared environment, generating an immediate savings in capital expenses. “It was quite a boon for us,” Rogers said. “We were able to reprogram the money in other areas.”

It also generated energy savings. Electricity consumption went from 28,000 watts per hour to 10,000, and energy consumed for cooling went from 95,000 BTUs to 36,000.

In further IT conservation, a logical storage unit pools storage so that excess capacity is not wasted by being dedicated to a single use. Total storage capacity was reduced by about 40 percent, from 111 terabytes to 68 terabytes, without degrading performance.

The second phase of the program, load balancing, is now in progress and is expected to be completed in October. This will provide self-service provisioning and reduce the number of systems administrators needed from 20 to five.

Configuration management is a key element in administrative efficiency and security. Limiting variation in the virtual machine pools and closely controlling who is allowed to make changes ensures a consistent and reliable stack, which improves stability and security.  This should improve the user experience, according to the Air Force.

The third phase of the program, called hyper convergence, will reduce latency and increase efficiency by moving to a nonblocking architecture, in which switches will handle maximum transfer rates from all ports. This phase is expected to be completed by the end of fiscal 2015 and could increase the capacity of the system by as much as 32 times.

In the final phase, the SCE will  focus on  software defined networking and  storage, which is expected to enable more efficient management through the abstraction of some  functions. Storage will be provided by random access memory rather than spinning disk drives, which Rogers said should improve efficiency by an order of magnitude and increase capacity without extending the infrastructure.

It is too early for a timetable on the final phase, Rogers said. “I know the technology we want to install, but I don’t know what the technology will be when I get there.” He said he hopes to begin work on specifications by mid-2015.

The key to a successful shared computing environment is strong leadership to maintain configuration control, Rogers said. In a shared environment, small changes have big effects, and without control, you are back to day-to-day firefighting.