You can't leak unclassified memos. They're ... they're unclassified.
You can "leak" anything. Podesta's emails were not classified either, but no one balked at calling them "leaked", right? You're playing a semantics game. A private conversation with the president of the US, when you are an employee of the executive branch of the federal government is automatically considered privileged and confidential, even if not officially classified. Most people kinda get this right off the bat. And even if not illegal, choosing to send the contents of said conversation through a third party to a news organization is at the very least "questionable behavior" for someone in Comey's position.
At the end of the day, Comey was fired because he could not be trusted to be an honest unbiased operator. Ironically, prior to his actual firing, pretty much everyone on both sides of the political aisle agreed on this. It was only afterwards that the whole "OMG! Let's pretend that he wasn't fired for his ridiculous behavior last year, but because of <somethingsomething> Russian Collusion!!!" narrative started.
Him leaking a "memo" (let's recall that it's not like he was taking notes during the meeting, and we at this point have no clue when he actually wrote this), really only serves to justify the decision to fire him. He couldn't actually be trusted. Not by the people of the country (on any part of our political spectrum). Not by the men and woman working under him in the FBI. And not by the President of the US. How the heck did anyone think he could remain in that position? The fact that the FBI was moving forward with the Russian meddling investigation certainly compounded the problem, but not for the reason being claimed. No one could possibly trust that any politically connected investigation he might run would be free of the same kind of political shenanigans as what happened last year. And no one could therefore trust the outcome of any investigation he was involved in. No matter which "side" of the issue you may fall on, Comey could not be allowed to remain.