Computer audit logs showing what occurred on a vote tabulation system that lost ballots in the November election are raising more questions not only about how the votes were lost, but also about the general reliability of voting system audit logs to record what occurs during an election and to ensure the integrity of results.

The logs, which Threat Level obtained through a public records request from Humboldt County, California, are produced by the Global Election Management System, the tabulation software, also known as GEMS, that counts the votes cast on all voting machines – touch-screen and optical-scan machines – made by Premier Election Solutions (formerly called Diebold Election Systems).

The logs are at the core of an investigation that the California secretary of state's office is conducting to determine why the GEMS tabulation system deleted 197 ballots from the tallies of one precinct in Humboldt County during the November 4 general election. But instead of providing transparency into what occurred on the system, the GEMS logs have so far only baffled state investigators. Deputy Secretary of State Lowell Finley has referred to the logs as "'Greek' to anyone other than a programmer."

A computer scientist who is a recognized expert on electronic voting machines says the logs are no clearer to him.

"These audit logs could give us some assurances [about an election] if they were genuinely designed so that a casual bystander could look at them and understand them," says Doug Jones, a University of Iowa computer scientist and former chairman of a board that examines and approves voting machines for use in Iowa. "[But] having them cryptic and obscure destroys the value in terms of election transparency."

Computer audit logs are supposed to track activity on a voting system to help officials investigate problems when they occur and ensure that no one tampers with the software. But the audit logs produced by Premier's GEMS system don't appear to provide any date or time stamps to indicate when events occurred. [Jones couldn't decipher the format of what appeared to be an event code in the logs, but reader Tony Gutierrez says it actually is a Unix timestamp. See his comment below this post.] But the audit logs produced by Premier's GEMS system don't record when files are intentionally deleted from the system or unintentionally erased, the most fundamental information an audit log should record.

Humboldt County officials deleted at least 27 batches of optical-scan ballots from GEMS during the November election process – a common practice when officials make a mistake and have to re-scan a batch – yet the audit log shows that only one batch was "aborted" (aborting a batch is a different procedure from deleting a batch, which might explain why the log recorded this action and not the others). Here's how the aborted batch (referred to as "Deck 132") appears in the log:

1225737079 Exception! "Aborted" (Deck 132 on 192.168.3.100) 1225737079 Lost deck! 132 on 192.168.3.100

Unfortunately, however, deck 132 is not the batch of ballots that went missing from the system. The batch that went missing is deck 0. Line 11 of the log shows when deck 0 was uploaded to the system, but there's no line showing when it was deleted. Premier told state officials a different log, called the poster log, records deletions, but state officials could find no evidence of deletions in that log either (see below for more on the poster log).

Humboldt County officials created three versions of the audit log, one designated "Election Night Audit Log," one designated "Post Election Audit Log" and a third labeled "Final Canvas Audit Log." Although at least half of the ballots were scanned into the system after election night, all three audit logs are identical. The Election Night Audit Log includes all of the ballots that were scanned into the system weeks later, suggesting that the system might not be able to produce "snapshot" reports after an election to reflect how the system looked at a certain point in time during the election.

The logs also don't distinguish between test ballots scanned into the machine before the election that were never counted in the final results, and official ballots cast during the election. This created some confusion for Humboldt officials when they found 17 mysterious ballots in their audit log that they couldn't account for (see line 3 of the log). They eventually determined that these were ballots scanned into the system during logic-and-accuracy tests before the election. The ballots were deleted automatically when the system was re-set for the election, although the audit log doesn't indicate that the system was ever re-set or that the test ballots were deleted. (See the bottom of this post for information about how to read the audit logs.)

The audit logs appear to record only limited types of events on the system and provide no comprehensive record that tracks every event performed by an election official.

Premier didn't respond to a query from Threat Level about the logs. But Jones said the Premier/Diebold system, as far as he knows, provides no single log file that chronologically lists all events in the life of an election.

Instead, he says, the system keeps "lots and lots of different logs" that appear to have been "independently designed by people who didn't talk to each other" and that are incomprehensible to anyone except the vendor. He assumes Premier has documentation explaining how to interpret the logs, but says if it does, the company doesn't share that information with election officials, making independent audits of a voting system difficult if not impossible.

"From the point of view of actually doing any forensics, it's a mess," Jones said. "Because you have to understand what all of the logs are saying, and all of the documentation to understand what they're saying are not public documents. I find that truly reprehensible. The idea that you can have this inscrutable document, but that you can't have any document to understand that document, is just totally nuts."

Even more troubling than the audit logs is another log referred to as a "status report by deck" (.pdf). This log report, which was created after the post-election canvas was completed, lists each batch of ballots, called decks, and does show the dates and times ballots were scanned. Unfortunately, it shows the wrong date and time for at least three decks.

The report, for example, indicates that ballots that were loaded to the system on November 3, were loaded on November 25 instead (see the first entry for deck 0 on page 6 under the heading "mail ballot precincts"). A different status report created on election night (.pdf) correctly shows this same batch of ballots scanned on November 3.

Batches that were loaded sequentially also show up in the status reports with out-of-sync dates and times. The first batch of "mail ballot precincts" ballots that was scanned on November 3 was scanned right before the batch that follows it. But in both reports, the first batch appears to have been scanned after the batch that follows it. The election night report shows the first batch being scanned about 23 minutes after the second batch, while the final canvas report shows the first batch being scanned three weeks after the second batch, all of which raises serious questions about the integrity of the logs and their ability to monitor what occurs during an election.

County election officials discovered several weeks after the election, and after they had certified the official election results to the secretary of state's office, that the system had deleted 197 ballots.

Humboldt uses Premier's central-count optical-scan system, and an investigation so far shows that the paper mail-in ballots were scanned properly by officials into the optical-scan system. Humboldt County provided Threat Level with a manual log that officials filled out by hand as they scanned the ballots, as well as a receipt printed from the machine after they were scanned.

The manual log shows that the 197 ballots were scanned by Elections Manager Kelly Sanders on November 1, three days before the election. The receipt from the scanner also shows the ballots in the system, although there's no date on the receipt – another problem for conducting audits.

The ballots even showed up in the status report (.pdf) printed from GEMS on election night. (See the 197 ballots that show up as "deck 0" on the first page of this report.)

But some time after election night, the tabulation software deleted the ballots. (A second report created after the election canvas was completed (.pdf) shows the 197 ballots missing.)

Premier attributed the problem to a programming error in GEMS that causes the first "deck" or the first batch of ballots counted by the software to be randomly deleted if a subsequent deck is intentionally deleted.

The GEMS system names the first batch of ballots "deck 0," with subsequent batches called "deck 1," "deck 2," etc. The system creates a "deck 0" for each ballot type that is scanned. Humboldt County has three types of ballots – absentee ballots, provisional ballots and "mail ballot precincts" (ballots from precincts that are so small they use only mail-in ballots) – although the audit log actually shows four "deck 0" files, because one belongs to ballots scanned during the pre-election logic-and-accuracy test.

Premier said that a programming error sometimes causes the "deck 0" to be erased if any subsequent deck involving the same type of ballots is erased. The absentee "deck 0," containing 197 ballots, is the one that disappeared from the system in Humboldt County.

County officials deleted 26 different decks of absentee ballots, any one of which might have caused the deck 0 to disappear from the system. (Note that this refers to "deleted" decks as opposed to deck 132, which was aborted. A deletion occurs after a batch of ballots is already scanned into the system, and officials discover a mistake and delete the deck in order to re-scan the ballots. An abortion occurs when an official opens a new file or deck but aborts it before scanning any ballots in the batch.)

Premier initially told Humboldt County Registrar of Voters Caroline Crnich that the deck 0 got deleted when she aborted deck 132. But this proved to be incorrect when Crnich realized she'd aborted deck 132 before election day, and deck 0 was still in the system on election night. County officials also deleted a deck from the mail ballot precincts category, yet the "deck 0" in that category did not disappear from the system, which either confirms Premier's explanation that the deletion problem occurs sporadically or indicates that Premier's explanation of the problem is wrong.

The state's investigation is ongoing and has expanded to two other California counties – Santa Barbara and San Luis Obispo – that use the same version of GEMS software that Humboldt uses. The state plans to release a report of its findings to the federal Election Assistance Commission, to be shared with other election jurisdictions around the country.

A spokeswoman for the secretary of state's office had only this to say about the investigation so far:

"Secretary Bowen is certainly concerned about Premier's carelessness with yet another elections product and thinks it's distressing that the company took virtually no action for years on this apparent defect. Secretary Bowen is talking with the company, county elections officials and others about how to prevent this problem from ever happening again in California.

One last point. I mentioned something about the poster log. When California state investigators expressed their concern that the GEMS audit logs didn't show any record of Humboldt officials deleting 27 decks, Tab Iredale, a programmer who helped develope the Premier/Diebold system, told Deputy Secretary of State Lowell Finley that he would find the deletions recorded in a second set of logs – the poster logs.

Here's a copy of the poster log. As you can see, it's just a listing of "starts" and "stops" that appears to record each time a new batch of ballots was scanned and posted to the system. The only deviation in the list is one line that reads "Recovering 0 job(s)," with no indication of what this might mean. There's no mention of the 27 decks being deleted.

Finley hasn't responded to a call asking for comment, but here's an e-mail he sent to Humboldt County Registrar of Voters Carolyn Crnich in December:

From: Finley, Lowell Sent: Friday, December 05, 2008 6:33 PM To: Crnich, Carolyn Subject: RE: Poster Logs Carolyn, Tab Iredale said these logs would reflect deletions of decks, but they sure don't look like it. The "Recovering 0 jobs" may be related to the aborted Deck 132, but none of the rest of it seems to correspond to the deletions you knowingly made. In any event, I would say this would qualify as the proverbial "Greek" to anyone other than a programmer. I hope you have a great weekend. Lowell Finley Deputy Secretary of State Voting Systems Technology and Policy California Secretary of State Debra Bowen

Premier became aware of the problem of deck 0 being deleted from GEMS in 2004 with version 1.18.19 of its software. As I wrote in a previous post, the company sent election officials an e-mail describing a workaround that they should perform, but didn't tell officials the reason for the workaround (i.e., that the system might randomly delete votes if they didn't perform the workaround). The company apparently fixed the issue in a later version of the software, but never told California state officials about the flaw so they could urge counties to upgrade to the new version or ensure that county officials knew about the workaround (Crnich wasn't aware of it until the investigation began.)

The problem with "deck 0" being deleted from the GEMS software is unique to Premier's central-count optical-scan system, since it appears to be the only Premier system that creates decks to tabulate votes. However, the basic problems with the audit logs and their lack of transparency in tracking every event that occurs on the tabulation system is likely a global problem in the GEMS design that would affect the ability to audit elections on any Premier voting system.

Jones, who has been examining Premier voting systems since they were first known in the late 90s under the name I-Mark Systems (before Diebold purchased the technology), said there's no reason to think the audit logs are any more comprehensive for Premier's other voting machines.

If anyone has information about the GEMS logs for touch-screen and precinct-based optical-scan machines, I'd be interested in hearing about it.

HOW TO READ THE AUDIT LOG: The GEMS audit log includes a left-hand column of numbers that is a Unix timestamp, indicating the date and time the event occurred. appears to be an event ID. Although the numbers are sequential as you go down the list, there are gaps in them, indicating perhaps other events occurring on the system in between these events, which are not recorded on this log.

The log refers to two different IP addresses ("connection on 192.168.3.100" and "connection on 192.168.3.101"). These are two separate optical-scan machines that were used to upload the scanned ballots to the tabulation server. "New deck 0," "new deck 1," indicates each time a file is created for a new batch of ballots. Once the batch is scanned, the system "commits" the deck to the system and indicates the number of ballots in the batch ("Commit Deck 0 Count 197").

HOW TO READ THE STATUS REPORT: The status report (.pdf) has three sections for the different types of ballots – absentee, provisional, and mail ballot precincts. Reading left to right, the "Deck ID" column lists the number of the deck, followed by the date and time the deck was uploaded to the system, followed by the number of ballots that were in that deck.

You'll notice there are gaps in the Deck ID. For example, decks 5, 6, and 10 are missing. These decks are among the 27 decks that had mistakes and were deleted before being re-scanned. When they were re-scanned, they were scanned with a new deck number. Deck 5, for example, was renamed deck 8 when it was re-scanned. Although the status report doesn't indicate that these decks were deleted and re-scanned, Humboldt officials happened to keep a manual log recording each deck that was deleted and the new deck number when it was re-scanned.

See also: