© Larry Harmon stands in his home in Kent, Ohio, on Aug. 4, 2017. Harmon was removed from the voting r... Image: Larry Harmon stands in his home in Kent, Ohio, on Aug. 4, 2017. Harmon was removed from the voting rolls after not voting in the…

Larry Harmon, 60, hadn't voted in a while when he drove to the high school in November 2015 to weigh in on a local referendum in Kent, Ohio. But he wasn't allowed to cast his ballot.

Thanks to six years of inactivity — and a single piece of unanswered mail asking him to confirm his voter registration — Harmon, now a plaintiff in a major voter purge lawsuit before the Supreme Court, was removed from Ohio's voter rolls.

"I've been paying my taxes, paying my property taxes, registering my car," he said. "All the data was there for (election officials) to know that I was there."

Harmon was a casualty of the latest voting battleground: How America's lists of registered, eligible voters are maintained.

The rolls are messy, due to deaths as well as those who move between states and neglect to unregister from their past place of residence. Both sides of the debate say something should be done to ensure greater accuracy.

But voting rights advocates have expressed increased anxiety over how some states, like Ohio, have aggressively pruned their voter rolls in the name of preventing widespread, yet still unproven, voter fraud. Those advocates say voter purges require a surgeon's scalpel and argue that aggressive purges suppress mostly minority and Democratic votes.

Thanks to President Donald Trump's voter fraud commission and a Justice Department memo signaling his administration's desire to take action on voter roll maintenance, they fear even more eligible voters will be at risk.

In June, the DOJ alarmed advocates by sending a letter to 44 states — those covered by the National Voter Registration Act, which the Justice Department enforces — asking for details on their purge processes.

"This is a prelude to setting up voter purging," said Vanita Gupta, who during the Obama administration was head of the DOJ's Civil Rights Division, which houses voting rights enforcement.

Still, conservatives say messy voter rolls create opportunities for fraud.

"The threat is, at least in considerable part, voter fraud," said Bob Popper, a senior attorney and director of the Election Integrity Project at Judicial Watch, a conservative group that sued to insist on Ohio's voter purge process.

The president has said the messy voter rolls are a sign of fraud, and must be investigated.

"When you look at the people that are registered — dead, illegal and two states and some cases maybe three states — we have a lot to look into," Trump told ABC News in January.

Why Getting It Right Is So Hard

Voter purges — what election officials call the removal of outdated registrations — have existed as long as elections have, and are required by federal law.

Clean, accurate voter rolls might sound like an obvious goal. It saves local authorities money while making election operations more efficient. But the country's voter rolls are typically a mess. A 2012 Pew survey found one in eight active registrations to be invalid or inaccurate. The rolls showed 1.8 million dead voters still on the books, while 2.75 million people had registrations in more than one state.

One reason? The decentralized, localized voter rolls are built for registration, not removal. Getting on the rolls at a new address is easier than removing your name from the county you just left, and different states and districts manage their rolls differently, too.

What's more, keeping voter rolls properly maintained is complicated, since election officials have to keep tabs on every voter who dies or moves. Thousands of voters have the same name and even the same birthday, and mistakes happen.

Florida Gov. Rick Scott went to vote in 2012 only to find that election officials believed he'd died because a voter named Richard E. Scott, born the same day as the governor, had passed away in 2006. The governor, whose full name is Richard L. Scott, was forced to cast a provisional ballot, something that can act as a fail-safe if an error has been made in certain circumstances.

On paper, it's easy for multiple voters to appear identical. For example, if election officials found "Michael Smith" had moved and registered in a neighboring state, they might believe a Michael Smith of the same name and birthday on their voter rolls to be the same person, and begin the purge process. But there are 25,280 registered voters with the name Michael Smith, and 18,397 of them, or roughly 73 percent, share a birthday with another Michael Smith, according to voter file data provided by TargetSmart and independently analyzed by the NBC Data Analytics lab. Take the name of the president's new chief of staff, John Kelly: There are 4,422 registered; 1,297 share a birthday.

It's one of the reasons critics have sharply questioned the plans of Trump's voter integrity commission, which is seeking to collect and analyze state voter data. The commission's co-chairman, Kansas Secretary of State Kris Kobach, built an opt-in system for states to share and compare voter roll data, the Interstate Crosscheck System. But Crosscheck regularly produces false matches: One study found that for every double vote found in the system, 200 people could be disenfranchised. Oregon and Florida opted out over problems like this.

Popper, of Judicial Watch, blamed messy voter rolls for lukewarm confidence in elections. Just 35 percent of voters said they were "very" confident in U.S. elections, with another 31 percent characterizing themselves as "somewhat confident" in a 2016 Gallup poll.

Indiana Secretary of State Connie Lawson, who sits on Trump's commission, recently purged nearly half a million voters in her state, or 10 percent of the voter rolls, after she was deputized by the state legislature to clean up the state's rolls following a legal challenge by conservative groups Judicial Watch and True the Vote.

"It really is about the competence of the election. I've never had someone call my office and say my record was removed erroneously," Lawson told NBC News. "I have had people call…after seeing a deceased neighbor in the poll book, and say 'Why aren't you keeping these lists correct?'"

Advocates agree that cleaner voter rolls are much needed, but argue that eligible voters must not be disenfranchised in the process.

"Our concern with voter purges is you may take people off inaccurately and not give them notice," said Dale Ho, director of the Voting Rights Project at the American Civil Liberties Union. If you're showing up on Election Day and trying to vote and being told, sorry you're not on the voter rolls, you've effectively been disenfranchised, through no fault of your own."

Thousands Knocked Off the Rolls

There's no way of knowing exactly how many people have been erroneously removed from records across the nation.

Purges are done by countless officials at the state and local level. There's no standardized way of keeping records and almost no way of telling eligible voters who are purged improperly from appropriate cuts. Wrongful purges are often not known until eligible voters are turned away at the polls, as Harmon was at his local high school when he tried to vote on a local marijuana referendum.

In Harmon's case, inactivity — namely, choosing not to vote in midterm elections of 2010 — triggered what's called a "confirmation process," wherein a letter was sent to his address asking him to confirm his voter registration. Harmon never saw it; he didn't vote in 2012 or 2014, either, and was taken off the rolls four years after that notice. The Supreme Court will decide this fall whether inactivity and nonresponse to a mailing count as evidence of ineligibility, which is a key to legal purging under the 1993 National Voter Registration Act.

© Voters cast ballots inside the Prairie Township Fire Station polling location in Holmesville, Ohio, ... Image: Voters Cast Their Ballots For The 2016 U.S. Presidential Election

After Harmon, who was backed by advocates like the American Civil Liberties Union and the liberal think-tank Demos, contested Ohio's purges in court, a federal judge ordered that voters purged for inactivity and failing to respond to a notice be allowed to cast provisional ballots if they still lived in the county. The court order allowed 7,500 ballots to be counted in the 2016 presidential election, a number Demos' attorney Cameron Bell said is just a fraction of the number of eligible voters who were purged but might not have voted.

What's more, advocates say, these purges disproportionately affect minorities.

Purges are "going to hit the communities of color the most. We have U.S. Census data that shows that minorities are represented in 85 of the most common 100 names," said Gupta, now president and CEO of the Leadership Conference on Civil and Human Rights.

Ohio is hardly alone in aggressive purges. Georgia is being sued over a similar process.

And in North Carolina, ahead of the 2016 election, a group of counties purged 6,700 voters in predominantly black, Democratic-leaning communities whose mail was returned as undeliverable. A judge mandated voters' registrations be restored and called it "insane."

Meanwhile in Florida, after state officials identified 180,000 voters suspected of being noncitizens in 2012 by running federal immigration system data against its voter rolls, local officials refused to conduct the purges, citing enormous error rates. Eighty-seven percent of the voters on the list were minorities.

The data used to purge voters is key. In 2012, Texas compared its voting records to national death data, repeatedly mistaking healthy Texans for dead Americans in other states. The state later apologized and abandoned the practice.

The Future of Voter Purges

When Trump's voter fraud commission sent a letter to secretaries of state requesting public voter data, the request was met with a flurry of headline-making refusals, as voting rights advocates and privacy advocates alike denounced the idea. For some, however, the Department of Justice's inquiry to states about their purge processes, sent on the same day, set off a different alarm.

"Normally, the Department of Justice would send a letter like this to a particular state it was investigating, based on a piece of evidence that the state was violating the law," said Loyola Law School Professor Justin Levitt, who had worked at the Department of Justice. "You wouldn't see this if they weren't anticipating litigation."

The Justice Department isn't alone in considering lawsuits to prompt states to purge voter rolls. Many of the most aggressive purges in recent years have come thanks to legal challenges from outside parties, and it's a trend that promises to continue.

© Larry Harmon sits in his home in Kent, Ohio, on Aug. 4, 2017. Harmon was removed from the voting rol... Image: Larry Harmon sits in his home in Kent, Ohio, on Aug. 4, 2017. Harmon was removed from the voting rolls after not voting in the last…

Judicial Watch and another conservative group, True the Vote, sued Ohio in 2012, arguing that three counties had more voters than there were voting-age residents, while 31 counties in Ohio had voter rolls with 90-100 percent of all voting-age residents. Ohio settled in 2014, vowing to double down on purging. In April, Judicial Watch threatened 11 states with similar litigation.

"I would expect a couple of lawsuits in the next couple of months," Popper said.

Harmon, the Ohio voter turned away at the polls, initially thought he'd done something wrong when he couldn't vote.

"I kind of blamed myself initially," he said. "I did a little Googling, and I found out about the purge and found out that if you hadn't voted in six years they'd take you off. Then I got a little mad."

Now, as his case heads to the Supreme Court, he said feels good that he's spoken up.

"I'm glad I stood up for myself and wrote a letter and complained about it. I think a lot of people just think it's not a big deal — if I don't use my 'right to bear arms,' they don't stop me from buying a rifle next year because I didn't buy one this year."