Flaw in vote-tallying software silently deletes votes

A flaw in software used to tally mail-in ballots resulted in the loss of 197 votes in one California precinct during last year’s general election, according to Secretary of State Debra Bowen.

The flaw was found in a version of software from Premier Election Solutions Inc. (formerly Diebold) of Allen, Texas, used to count mail-in ballots that had been optically scanned. The problem was found in Global Election Management System (GEMS) version 1.18.19, and only affects precincts using the system’s Central Count server.

The software was corrected in later versions, but users of the affected software were not fully informed of the problem even though the company knew of it in 2004, Bowen said in a report filed with the U.S. Election Assistance Commission (EAC).

“The software error silently deletes all tallied votes from the first batch or ‘deck’ of optical scan paper ballots after they have been scanned into GEMS,” Bowen wrote. “The deletion results whenever...at any point after the first deck of voted ballots (automatically named “Deck 0” in GEMS 1.18.19) is scanned into the GEMS database, the Central Count Server window is closed and reopened; and the GEMS operator deletes any subsequent deck of ballots because a problem is encountered.”

Premier pointed out that only three counties in the country -- Humboldt, Santa Barbara and San Louis Obispo, all in California -- use the combination of this specific version of GEMS and the Central Count server. The company also said it notified customers of the problem and provided a workaround procedure to mitigate the problem.

Additional problems also were discovered in the audit logs of the software that failed to record some events and allowed some records to be removed from the system.

The report will be released March 17 at a public hearing in Sacramento, Calif.

The error was discovered after a volunteer watchdog group digitally scanned ballots cast in the Humboldt County precinct to enable an independent tally of votes. The unofficial tally, which was conducted with the cooperation of county election officials, came up with a total of 216 more votes than the official one. The bulk of these were tracked to a 197-vote discrepancy in the counting of mail-in ballots that had been optically scanned and counted with the GEMS software.

An investigation by election officials identified the problem that allowed some batches or decks of votes to be deleted.

“Diebold knew of this serious software error no later than October 2004,” Bowen wrote. “The company, however, did not notify the Election Assistance Commission, the National Association of State Election Directors [NASED] or the California secretary of state.”

Premier spokesman Chris Riggall said that “when this matter arose five years ago, there was no mechanism, procedure or mandate for reporting issues of this nature to the California Secretary of State’s office. Premier adhered to the reporting protocols that existed at the time.”

The error was corrected in GEMS version 1.18.24, but users of earlier versions were only given a work-around in an October 2004 e-mail.

“It is very important that you follow these instructions,” Diebold said in the e-mail copied in Bowen’s report. “ISSUE: When running Gems 1.18.19.0 and processing ballots with the Central Count Server an issue exists with correctly sorting committed decks, in some reports, and also deleting other decks under certain conditions, when ‘deck 0’ has not been deleted.

“RESOLUTION: When the election is invoked and there has been no Central Count ballot processing ever done in the database then start the Central Count server and process a ‘Start’ card and then immediately afterwards an ‘Ender’ card. This will commit deck 0 without any ballots and allow the deletion of the committed deck 0 from the database. You should delete Deck 0. This must be done as the first action after starting Central Count.”

Humboldt County’s chief election officer, who assumed the job in 2007, had not seen the e-mail. But Riggall said that Premier’s response was not limited to the e-mail.

“In addition to an e-mail provided to counties, this topic was discussed regularly and repeatedly with effected customers on conference calls, at conferences and in one-on-one conversations between Premier associates and election administrators,” he said. After the problem was corrected, “we repeatedly encouraged all our California customers to upgrade to this enhanced GEMS release. The three counties in question elected not to upgrade and, in fact, sought and received permission from the California Secretary of State to continue to use 1.18.19.”

During the investigation of the deleted votes, problems also were found in the software’s audit logs, which apparently did not record deletions, provided wrong date and time-stamps for some events and allowed some crucial logs to be deleted. The “clear” button at fault for this was removed in next version of the software.

Because of these issues, Bowen said that the software was in violation of federal standards administered by NASED at the time GEMS was certified. Riggall dismissed this claim as “Monday morning quarterbacking of a system that was, in fact, fully certified for use by both NASED and the California Secretary of State. Federal and state certification authorities had full access to the features, components and source code of 1.18.19 and, clearly, found it to be compliant with relevant standards.”