With Russia already meddling in 2016, a ragtag group of obsessive tech experts is warning that stealing the ultimate prize in —victory on November 8— would be child’s play.
When Princeton professor Andrew Appel decided to hack into a voting machine, he didn’t try to mimic the Russian attackers who hacked into the Democratic National Committee’s database last month. He didn’t write malicious code, or linger near a polling place where the machines can go unguarded for days.
Instead, he bought one online.
With a few cursory clicks of a mouse, Appel parted with $82 and became the owner of an ungainly metallic giant called the Sequoia AVC Advantage, one of the oldest and vulnerable, electronic voting machines in the United States (among other places it’s deployed in Louisiana, New Jersey, Virginia and Pennsylvania). No sooner did a team of bewildered deliverymen roll the 250-pound device into a conference room near Appel’s cramped, third-floor office than the professor set to work. He summoned a graduate student named Alex Halderman, who could pick the machine’s lock in seven seconds. Clutching a screwdriver, he deftly wedged out the four ROM chips—they weren’t soldered into the circuit board, as sense might dictate—making it simple to replace them with one of his own: A version of modified firmware that could throw off the machine’s results, subtly altering the tally of votes, never to betray a hint to the voter. The attack was concluded in minutes. To mark the achievement, his student snapped a photo of Appel—oblong features, messy black locks and a salt-and-pepper beard—grinning for the camera, fists still on the circuit board, as if to look directly into the eyes of the American taxpayer: Don’t look at me—you’re the one who paid for this thing.
Appel’s mischief might be called an occupational asset: He is part of a diligent corps of so-called cyber-academics—professors who have spent the past decade serving their country by relentlessly hacking it. Electronic voting machines—particularly a design called Direct Recording Electronic, or DRE’s—took off in 2002, in the wake of Bush v. Gore. For the ensuing 15 years, Appel and his colleagues have deployed every manner of stunt to convince the public that the system is pervasively unsecure and vulnerable.
Beginning in the late ’90s, Appel and his colleague, Ed Felten, a pioneer in computer engineering now serving in the White House Office of Science and Technology Policy, marshaled their Princeton students together at the Center for Information Technology Policy (where Felten is still director). There, they relentlessly hacked one voting machine after another, transforming the center into a kind of Hall of Fame for tech mediocrity: reprogramming one popular machine to play Pac-Man; infecting popular models with self-duplicating malware; discovering keys to voting machine locks that could be ordered on eBay. Eventually, the work of the professors and Ph.D. students grew into a singular conviction: It was only a matter of time, they feared, before a national election—an irresistible target—would invite an attempt at a coordinated cyberattack.
The revelation this month that a cyberattack on the DNC is the handiwork of Russian state security personnel has set off alarm bells across the country: Some officials have suggested that 2016 could see more serious efforts to interfere directly with the American election. The DNC hack, in a way, has compelled the public to ask the precise question the Princeton group hoped they’d have asked earlier, back when they were turning voting machines into arcade games: If motivated programmers could pull a stunt like this, couldn’t they tinker with the results in November through the machines we use to vote?
This week, the notion has been transformed from an implausible plotline in a Philip K. Dick novel into a deadly serious threat, outlined in detail by a raft of government security officials. “This isn’t a crazy hypothetical anymore,” says Dan Wallach, one of the Felten-Appel alums and now a computer science professor at Rice. “Once you bring nation states’ cyber activity into the game?” He snorts with pity. “These machines, they barely work in a friendly environment.”
The powers that be seem duly convinced. Homeland Security Secretary Jeh Johnson recently conceded the “longer-term investments we need to make in the cybersecurity of our election process.” A statement by 31 security luminaries at the Aspen Institute issued a public statement: “Our electoral process could be a target for reckless foreign governments and terrorist groups.” Declared Wired: “America’s Electronic Voting Machines Are Scarily Easy Targets.”
For the Princeton group, it’s precisely the alarm it has been trying to sound for most of the new millennium. “Look, we could see 15 years ago that this would be perfectly possible,” Appel tells me, speaking in subdued, clipped tones. “It’s well within the capabilities of a country as sophisticated as Russia.” He pauses for a moment, as if to consider this. “Actually, it’s well within the capabilities of much less well-funded and sophisticated attackers.”
In the uproar over the DNC, observers have been quick to point out the obvious: There is no singular national body that regulates the security or even execution of what happens on Election Day, and there never has been. It’s a process regulated state by state. Technical standards for voting are devised by the National Institute of Standards and Technology and the Election Assistance Commission—which was formed after the disputed 2000 presidential election that hinged on faulty ballots—but the guidelines are voluntary. (For three years the EAC limped on without confirmed commissioners—an EAC commissioner stepped down in 2005, calling its work a “charade”). Policy on voting is decided by each state and, in some cases, each county—a system illustrated vividly by the trench warfare of voter ID laws that pockmark the country. In total, more than 8,000 jurisdictions of varying size and authority administer the country’s elections, almost entirely at the hands of an army of middle-age volunteers. Some would say such a system cries out for security standards.
If such standards come to fruition, it will be the Princeton group—the young Ph.D.’s who have since moved on to appointments and professorships around the country—and their contemporaries in the computer science world who suddenly matter.
The Princeton group has a simple message: That the machines that Americans use at the polls are less secure than the iPhones they use to navigate their way there. They’ve seen the skeletons of code inside electronic voting’s digital closet, and they’ve mastered the equipment’s vulnerabilities perhaps better than anyone (a contention the voting machine companies contest, of course). They insist the elections could be vulnerable at myriad strike points, among them the software that aggregates the precinct vote totals, and the voter registration rolls that are increasingly digitized. But the threat, the cyber experts say, starts with the machines that tally the votes and crucially keep a record of them—or, in some cases, don’t.
Since their peak around 2007, voting districts have begun to rely less on the digital voting machines—a step in the right direction, as states bolt for the door on what the programmers describe as a bungled, $4 billion experiment. Instead, rushing to install paper backups, sell off the machines and replace them with optical scanners—in some cases, ban them permanently for posterity. But the big picture, like everything in this insular world, is complicated. As the number of machines dwindle—occasioned by aging equipment, vintage-era software that now lacks tech support, years without new study by the computer scientists, and a public sense that the risk has passed—the opportunities for interference may temporarily spike. Hundreds of digital-only precincts still remain, a significant portion of them in swing states that will decided the presidency in November. And, as the Princeton group warns, they become less secure with each passing year.
In American politics, an onlooker might observe that hacking an election has been less of a threat than a tradition. Ballot stuffing famously plagued statewide and some federal elections well into the 20th century. Huey Long was famously caught rigging the vote in 1932. Sixteen years later, 1948 saw the infamous “Lyndon Landslide,” in which Johnson mysteriously overcame a 20,000 vote deficit in his first Senate race, a miracle that Robert Caro reports was the almost certain result of vote rigging. But even an unrigged election can go haywire, as the nation learned in horror during the Florida recount in 2000, when a mind-numbingly manual process of counting the ballots left a mystery as to which boxes voters had punched—giving the nation the “hanging chad,” and weeks of uncertainty about who won the presidency.
In some ways, the country’s response was suggestive of the real crime committed in Florida: Not inaccuracy, but anxiety. Congress’s solution was to pass the Help America Vote Act in 2002, a nearly $4 billion federal fund meant to incentivize states to upgrade their voting machines. It worked. All 50 states took the money. Requirements included upgrading voter registration methods and making polls disability-friendly, but Section 102 provided funds specifically allocated for replacing outdated voting machines; almost universally, “upgrade” meant a new, computerized touch-screen voting machine. By 2006, states had spent nearly $250 million on new machines with Section 102 funds. In Pennsylvania, the funds purchased 20,597 new machines—around 19,900 of which were digital touchscreens. Some, like the Diebold TSX, Advanced WINvote, the ES&S iVotronic, and a variant of Appel’s AVC Advantage—the Sequoia Edge—would be the same models to come under scrutiny by cybersecurity experts and academics. Thousands of touchscreen DREs were similarly sold in state contracts. Between Election Day 2000 and the HAVA cutoff in 2006, the stock prices of the major companies soared.
Kiyomi Fukushima (right) tries out an iVotronic electronic voting machine on display at Leisure World retirement community September 23, 2002 in Seal Beach, California. This is the first look for voters at three electronic systems vying to become the voting system of choice in California for the 2004 election cycle. The Registrar of Voters is holding a series of meetings over the next three weeks to show the systems to the public | David McNew/Getty Images
The appeal of such machines seemed plain: Voting was crisp, instantaneous, logged digitally. To state officials—and, at first, voters—the free federal money seemed like a bargain. To computer scientists, it seemed like a disaster waiting to happen. Wallach remembers when he testified before the Houston City Council, urging members not to adopt the machines. “My testimony was: ‘Wow, these are a bad idea. They’re just computers, and we know how to tamper with computers. That’s what we do,’” Wallach recalls. “The county clerk, who has since retired, essentially said, ‘You don’t know anything about what you’re talking about. These machines are great!’ And then they bought them.”
Almost from the day they were taken out of the box, the touch-screen machines demonstrated problems (the same companies had a much better track record with Optical Scan machines). During the primaries in Florida in 2002, some machines in Miami-Dade malfunctioned and failed to turn on, resulting in hourslong lines that locked out untold numbers of voters—including then-gubernatorial candidate Janet Reno. That year, faulty software (and an administrator oversight) on Sequoia models led to a fourth of votes initially omitted during early voting in Albuquerque’s Bernalillo County. In Fairfax County, Virginia, an investigation into a 2003 school board race found that a vote was subtracted for every 100 votes cast for one of the candidates on 10 machines. With margin sizes small enough to be noticed, local elections were vaulted into the forefront of these debates; Appel later found himself issuing expert testimony for a tiny election for the Democratic Executive Committee in Cumberland County, New Jersey, where a candidate lost by 24 votes. The margin was small enough that the losers sued, and called 28 voters as witnesses—who each swore they voted for them. The machine in use was a Sequoia AVC Advantage.
“Wow, these are a bad idea. They’re just computers, and we know how to tamper with computers” — Dan Wallach, Felten-Appel alums andcomputer science professor
Cybersecurity researchers flocked to study the machines, but they say they were faced with an uncompromising adversary: the voting machine companies, which viewed the code of the machines as intellectual property. Until 2009, two companies, Diebold and ES&S, controlled the lion’s share of the voting machine market. The accreditation process is equally narrow: Since 1990, a voluntary federal accreditation process has certified voting technology, a system that has come under fire for its lack of transparency. The laboratories (“Independent Testing Authorities”) which conduct the certification reviews are typically paid by the manufacturers, and are usually required to sign nondisclosure agreements. In 2008, five labs were accredited; one was suspended that year for poor lab procedures, and another temporarily suspended for insufficient quality control.
State authorities can typically request these lab reports, as Kathy Rogers of ES&S reminded me in an email. (“For security reasons we did not make that code widely available to just anyone and everyone who simply wanted a copy for their own purposes. We truly have nothing to hide.”) But Appel, the Princeton group and others in cybersecurity have insisted that such measures—which they deem “security through obscurity”—pale to the types of rigorous testing that would result from releasing the code to the public or academics. One of the companies, Sequoia, later acquired by Dominion, once threatened Princeton’s Felten and Appel with legal action if they attempted to examine one of their models.
Election officials have sometimes complained that the lab reports they do receive lack vital detail, and information from the labs, bound by the NDAs, can be unforthcoming. In 2004, when the California Secretary of State Kevin Shelley—in charge of overseeing the state’s elections—asked one of the five laboratories for more information on the testing of machines, he was stonewalled, and told by a researcher, “We don’t discuss our voting machine work.” Because of a flood of machines introduced to the market after HAVA, the 2002 accreditation standards are the ones that matter—the same process that approved touch-screen Diebold machines that had supervisor passcodes of “1111” in order to access the voting system. Shelley later banned Diebold TSX machines, calling Diebold’s conduct “deceitful.”
In 2003, an employee at Diebold mistakenly left 40,000 files containing code for the Diebold AccuVote TS, one of the most widely used machines on the market, on a publically viewable website. The computer scientists moved in, and one of the early and formative papers was published on the subject, co-authored by Wallach and led by Johns Hopkins’ Avi Rubin. Its findings were devastating: The machine’s smartcards could be jerry-rigged to vote more than once; poor cryptography left the voting records file easy to manipulate; and poor safeguards meant that a “malevolent developer”—an employee inside the company, perhaps—could reorder the ballot definition files, changing which candidates received votes. The encryption key, F2654hD4, could be found in the code essentially in plain view; all Diebold machines responded to it. (Rubin later remarked that he would flunk any undergrad who wrote such poor code.) “We read the code, and found really, really bad problems,” Wallach tells me, sitting at his Houston dining table. He catches himself. “Actually, let me change that,” he says. “We found unacceptable problems.” Diebold dismissed the report, responding that the code was obsolete, and the study’s findings thusly moot. But the 2003 report catalyzed a small movement: In CompSci departments across the country, vote hacking became a small, insular civic code of honor. Felten’s group at Princeton led the pack, producing some of the most important papers throughout the 2000s.
The red ‘Vote’ light on an iVotronic electronic voting machine is chosen during a demonstration at Leisure World retirement community September 23, 2002 in Seal Beach, California | David McNew/Getty Images
By the following year, professors in and around the Princeton group began the work of unwinding what they viewed as a 50-state debacle. Felten and Appel shared a taste for gallows humor and a flair for promotion. Felten took to blogging, and started a tradition: Each election, he snapped a photo standing alone with unguarded voting machines days before the election. In another study, the Sequoia AVC Edge was infected with malware that allowed it to do nothing but play Pac-Man; the students pulled off the feat without breaking the machines “tamper-proof” seals, and decorated the machine with Pac-Man logos. The team tore through topics including source code review of the larger Diebold voting system; advising election officials on security measures without new hardware; and designing malware for the Sequoia AVC Advantage that Appel had purchased, using a technique called a Return-Oriented Program. In less than a minute, they infected a Diebold machine with self-duplicating code, spreading from machine to machine through an administrator card, and programmed it to swing an election for Benedict Arnold over George Washington.
“We read the code, and found really, really bad problems. Actually, let me change that, we found unacceptable problems” — Dan Wallach, Felten-Appel alums andcomputer science professor
The latter hack was the result of a curious and enigmatic email, when Felten received a message from an anonymous source, presumably with ties to the voting machine industry. Diebold’s response to the Rubin and Wallach study was brittle and evasive; the source wanted to give Felten a Diebold TS machine—the same one whose code had leaked in the study. Studying the machine itself would offer an unmissable opportunity—Felten put his grad students, Feldman and Halderman, then 25 years old, in charge of the effort. One night in April 2006, Halderman drove to New York City, and double-parked his car, lights blinking, in front of a hotel just a few blocks from Times Square. Halderman jogged into an alleyway, where his source stood patiently, dressed in a charcoal colored trench coat and wielding a black canvas bag. After a few terse formalities, he handed Halderman the bag with the machine inside. Halderman never saw the man again. (“There’s a lot of cloak and dagger in election security,” Halderman would tell me later.)
Throughout the summer of 2006, Feldman and Halderman set themselves to work in the basement of an academic building. Fearing retribution or a lawsuit, they didn’t tell their colleagues in the department of their project. From noon until midnight, the two students met on the humid Princeton quad, and decamped to a claustrophobic, eggshell anteroom—enough space for a small table and two uncomfortable foldout chairs—and pored through reams of code and programming under the fluorescent lighting of the windowless room. At the center of the table was the subject of years of mystery: The squat, beige monitor of the Diebold TS. The authors would later describe the project as the first rigorous analysis of a physical touch-screen DRE—supposedly the kind of testing it would have received in one of the accredited labs.
When they were finished, they had another paper’s worth of findings, and the most comprehensive understanding of how Diebold’s machines worked. “We found the machine did not have any security mechanisms beyond what you’d find on a typical home PC,” Halderman told me. “It was very easy to hack.” Studying with Felten, Halderman had learned a key phrase—“Defense in Depth,” meant to describe a system with various rings of security. Halderman joked that the model should more aptly be called “Vulnerability in Depth,” so numerous were the entry points they discovered. Later, they found the key that opened the Diebold AccuVote TS was a standard corporate model, reproduced for minibars and other locks, available online. When their report revealed this detail, a commonplace reader found a picture of the key, filed down a blank from ACE Hardware and sent a copy to Feldman and Halderman as a souvenir (who then tested the key—it worked). That year, 10 percent of registered voters alone used the AccuVote TS to vote.
None of these breakthroughs were lost on states that had bought the machines, officials who were keeping an eye on academic reports. Felten would later write that the vulnerabilities in the Diebold machine they tested likely could not be rectified without fully redesigning the machine; but the solution for state officials was simple. If they could include a paper trail—a voter-verified paper receipt that printed alongside the digital vote—the electronic tally could, in theory, be cross-tested for accuracy. In December 2003, Nevada became the first state to mandate that voter verified printouts be used with digital touch screens. A wave of states followed.
But the tipping point came in 2006, when a major congressional race between Vern Buchanan and Christine Jennings in Florida’s 13th District imploded over the vote counts in Sarasota County—where 18,000 votes from paperless machines essentially went missing (technically deemed an “undervote”) in a race decided by less than 400 votes. Felten drew an immediate connection to the primary suspect: The ES&S iVotronic machine, one of the many ordered in Pennsylvania after they deployed their HAVA funds. Shortly after the debacle, Governor Charlie Crist announced a deadline for paper backups in every county in Florida that year; Maryland Governor Bob Erlich urged his state’s voters to cast an absentee ballot rather than put their hands on a digital touch screen—practically an unprecedented measure. By 2007, the touch screens were so unpopular that two senators, Bill Nelson of Florida and Sheldon Whitehouse of Rhode Island, had introduced legislation banning digital touch screens in time for the 2012 election.
Precincts today that vote with an optical scan machine—another form of DRE that reads a bubble tally on a large card—tend not to have this problem; simply by filling it out, you’ve generated the receipt yourself. But that doesn’t mean the results can’t still be tampered with, and Felten’s students began writing papers that advised election officials on defending their auditing procedures from attempted manipulation.
Each state bears the scars of its own story with digital touch screens—a parabola of havoc and mismanagement that has been the 15-year nightmare of state and local officials. The touch screens peaked in 2006, touching nearly 40 percent of registered voters; in 2016, most voters will use some combination of paper, optical scan or paper backup. In 2013, Maryland sped up its wind-down process, pushing through a transition to optical scans for use in the 2016 election. So did Virginia, which has rushed to phase out as many as possible in time for 2016—and later passed legislation to ban them permanently by 2020, just for good measure.
The Virginia ban was the quixotic crusade of one computer science expert in the private sector, Jeremy Epstein. In 2002, Epstein walked into the elections office in Fairfax, Virginia, to complain about the poor design of the touch screens—a WINVote model—and walked out with a mission to get them barred from the state. The machines were connected to Wi-Fi—vulnerable to “anyone who wanted to could hack them from the comfort of their car out in the parking lot,” Epstein told me. An investigation later revealed that the WINVote’s encryption key was “abcde.” The machines were certified in 2003, running on a version of Windows from 2002, and hadn’t received an update since 2005.
Thirteen years later, Virginia announced its ban. “If these machines and elections weren’t hacked,” Epstein later told me, a credo he’s said for years, “it was only because no one tried.
In 2001, the notion of foreign vote hacking felt like a far-fetched warning from a far-off time—it would be years, for instance, before North Korean agents would hack a company like Sony, or the Chinese would break into the federal government’s personnel files. Citizen activists who had exposed the Diebold code leak and joined the counterreformation for paper ballots were concerned, but primarily about domestic hacking. Liberals tended to see the corporate voting machine companies as a threat to fair elections. Conservatives tended to see the incompetence of poorly designed machines as a threat to normalcy.
Today, Halderman reminds me, “the notion that a foreign state might try to interfere in American politics via some kind of cyber-attack is not far-fetched anymore.”
The Princeton group has no shortage of things that keep them up at night. Among possible targets, foreign hackers could attack the state and county computers that aggregate the precinct totals on election night—machines that are technically supposed to remain non-networked, but that Appel thinks are likely connected to the Internet, even accidentally, from time to time. They could attack digitized voter registration databases—an increasingly utilized tool, especially in Ohio, where their problems are mounting—erasing voters’ names from the polls (a measure that would either cause voters to walk away, or overload the provisional ballot system). They could infect software at the point of development, writing malicious ballot definition files that companies distribute, or do the same on a software patch. They could FedEx false software to a county clerk’s office and, with the right letterhead and convincing cover letter, get it installed. If a county clerk has the wrong laptop connected to the Internet at the wrong time, that could be a wide enough entry window for an attack.
“No county clerk anywhere in the United States has the ability to defend themselves against advanced persistent threats,” Wallach tells me, using the parlance of industry for highly motivated hackers who “lay low and stick around for a while.” Wallach painted an unseemly picture, in which a seasoned cyber warrior overseas squared off against a septuagenarian volunteer. “In the same way,” continues Wallach, “you would not expect your local police department to be able to repel a foreign military power.”
In the academic research, hacks of the machines are far more pervasive; digitized voting registrations or tabulation software are not 10 years old and running on Windows 2000, unlike the machines. Still, they present risks of their own. “There are still plenty of computers involved” even without digital touch screens, says Appel. “Even with optical scan voting, it’s not just the voting machines themselves—it’s the desktop and laptop computers that election officials use to prepare the ballots, prepare the electronic files from the OpScan machines, panel voter registration, electronic poll books. And the computers that aggregate the results together from all of the optical scans.”
“If any of those get hacked, it could could significantly disrupt the election.”
The digital touch screens, even with voter verified paper trail, will still be pervasive this election; 28 states keep them in use to some degree, including Ohio and Florida, though increasingly in limited settings. Pam Smith, the director of Verified Voting—a group that tracks the use of voting equipment by precinct in granular detail—isn’t sure how many digital touch screens are left; no one I spoke with seemed to know. Nor is it clear where they’ll be deployed, a decision left up to county administrators. Smith confirms that after 2007, the number of states that adopted the machines plateaued, and has finally begun to shrink. The number of states using paperless touch screens—and nothing else—is five: South Carolina, Georgia, Louisiana, New Jersey and Delaware. But the number of states with a significant number of counties with the easily hacked machines is much larger, at 13, including Indiana, Virginia, and Pennsylvania. For hacking purposes, there’s little difference: In a close election, only a few precincts with paperless touch screens would be required to deflate vote totals, says Appel, even if the majority of counties are still in the Stone Age. Many of Felten’s mad-scientist experiments were designed to metastasize the nefarious code once it gained entry into a machine system.
The move away from electronic voting is a positive one, the professors say; the best option for election security are the optical scans. “Although the optical scan ballots are counted by the computer in the OpScan machine—which you can’t trust—you can trust the pile of ballots that accumulate in the ballot box, marked by users with their own hands,” Appel tells me. With the right auditing policies, “you can recount or do a statistical sample of the ballot boxes to make sure there aren’t cheating computers out there.”