How county prepared for DDOS attacks at GOP convention
Connecting state and local government leaders
Concerned about being targeted by denial-of-service attacks while hosting the GOP convention, the Hillsborough County, Fla., Sheriff’s Office turned to security-as-a-service.
There are a variety of techniques for DOS attacks. The simplest — and easiest to counter — uses brute force, with one machine directing a flood of packets at a target. Weaponized tools such as the Low Orbit Ion Cannon that send out floods of TCP or UDP packets can automate the process, but the attacks and originating IP addresses can easily be identified and blocked.
Distributing the attack over a variety of machines and locations that each generate smaller volumes of malicious traffic can make the streams harder to identify and block. There also are tools that can use specially crafted packets and types of traffic that can overwhelm a server with lower volumes of traffic.
In late August, Tampa was facing two approaching storms. The first one, Isaac, which eventually struck New Orleans as a Category 1 hurricane, was not a real problem for Florida.
“It was a couple of days of high wind and lots of rain,” said Christopher Peek, CIO of the Hillsborough County Sheriff’s Office. “In Florida that happens frequently enough that it was a pretty routine event.”
Less routine was the Republican National Convention, which was scheduled to be held in Tampa Aug. 27-30. There not only were physical security requirements for this high-profile event, but cybersecurity to be considered as well, Peek said.
Related story:
Anonymous lures unwitting users into online campaign
“We had seen some other government entities come under denial-of-service attacks at similar events,” he said. The most recent such incident was during the spring NATO Summit in Chicago, when groups affiliated with the hacker collective Anonymous succeeded in blocking access to NATO and Chicago Police Department websites on May 20.
“We knew we were just as vulnerable as anybody, so we started to look around for what we could do to prevent it from becoming an issue,” Peek said.
The sheriff’s department settled on a hosted service from Black Lotus Communications that uses a proprietary technology called Human Behavior Analysis to spot and block malicious traffic even when it is coming in from distributed locations slowly enough to avoid detection by most filters.
As it turned out, Hurricane Isaac managed to knock out the first day of the GOP convention, but hackers were not a problem for police. “Knock on wood, it’s a service we haven’t needed,” Peek said. But the department now is maintaining the service as part of its ongoing network security. Like any kind of security, “if everything goes well, most people never know it’s there. But when you do need it you’re glad you have it.”
Hillsborough County is home to Tampa and St. Petersburg and covers about 1,100 square miles on Florida’s central Gulf coast. The Sheriff’s Office network connects 35 facilities and about 4,200 users and incorporates a lot of wireless connections. There are about 1,600 regular mobile users and another 200 that use mobile devices occasionally, Peek said.
Mobile networking has not been a security problem, he said. “We pretty much handle that in stride,” with wireless VPNs and two-factor authentication to secure network access. “In the old days you were rolling your own,” piecing together security components from a variety of vendors. “Over the years it has become a lot easier,” with mature solutions for identity and access control.
But denial-of-service attacks against public-facing resources such as the department’s website was another issue. Just about anything that makes a server or its contents unavailable, including damaging files or software, can result in a denial of service. But the typical DOS attack overloads the application, the server or the pipes, feeding it with more requests or traffic than it can handle. Even if the equipment and network continue to operate, legitimate traffic can be crowded out, making the resources unavailable. Such attacks usually do no lasting damage and are effective only while they are actively under way.
But being taken “offline” during a high-profile event can be embarrassing for the victim and give publicity to the attacker claiming credit. It also can deny the public access to important services and information.
The Hillsborough County Sheriff’s Office began looking in June for a hosted service to protect its website. There were two primary reasons for seeking a service rather than a tool for countering DDOS. One was a desire to avoid up-front capital costs for hardware and software to protect sites.
The second was bandwidth. If the protection was placed at the department’s perimeter, the pipes coming to the department still could be vulnerable to floods of traffic.
“What we found was that even with the tools in place, you were still going to have your bandwidth overwhelmed,” Peek said. Paying for the capacity needed to protect against such an attack would be expensive. The answer was to identify and halt the malicious traffic upstream, before it hit the department or its carrier. “We found it was far more cost-effective to procure a service.”
Hillsborough County was unusual in seeking this type of protection in advance of a high-risk situation, said Jeffrey Lyon, CEO of Black Lotus.
“Governments usually have a head-in-the-sand attitude about defense,” Lyon said. “They don’t typically look for leading edge technologies.”
Black Lotus protects its customers by acting as a carrier, announcing the customer’s IP status. “The traffic first goes to the Black Lotus network as part of its routed path,” Lyon said. Traffic is monitored well upstream of the customer, analyzing the header of one out of every 1,000 packets. If there is evidence of an attack, the malicious flow is directed to the system for deeper analysis and filtering.
One tenth of one percent is not a lot of traffic to look at, but Lyon says it is “entirely adequate for a denial-of-service attack.” He said the company has had very few inquiries about false positives, situations in which legitimate traffic is mistakenly blocked. “I’d put our accuracy level at four or five nines,” (up to 99.999 percent) he said.
It turned out that opting for a service as opposed to a point solution was not necessarily going to be inexpensive. “It’s not a cheap service,” Peek said. “We were seeing quotes of from $8,000 to $15,000 a month.”
Getting a budget for IT security is often a problem, especially in government, because it is difficult to build a business case based on a return on investment, he said. “It’s a service you hope you never need, so it’s difficult to get someone to go along with the funding.”
Fortunately, Black Lotus not only had a service that would work for the department, it also was flexible on the price. “They were willing to look at our situation and work with us to make it affordable. They really worked with us and got it up and running in a short time.”
Black Lotus did not disclose the terms of its deal with Hillsborough County, but it offers plans for its hosted service beginning at $1,000 a month. The department made its decision to go with the service in mid-July and had it running by end of the month. Although the impetus for the protection was the GOP convention, the department has maintained the service.
“We were willing to look at it as a long-term service,” Peek said. “Based on their flexibility we felt it was a worthwhile tool to keep around.”
Peek’s advice on selecting a security service is to know what you need and shop around for the right providers. “Research and understand what you are trying to protect yourself against, and make sure the solution is a match.”