Yeah, remember that your power ups will slow down later on if you don't diversify. If you only use Destruction, it's going to initially shoot up, yes. And that's going to get you damage increases. But you go from 25 to 35 WAY faster than you'll go from 70 to 71. So you end up seeing a plateau in your power (though your actual level ups should be fairly constant, since higher skill-ups provide more experience).
By investing in a few skills, you won't see such a huge damage spike. But you'll see a more constant increase.
Personally, I would spike up Destruction initially, but then start trying to approach encounters strategically. That way, you still have high-damage spells to fall back on, but you'll also take other skills up. And don't discount some spells. Fury, for instance, is very nice if you can hit a group with it (especially from ranged before they attack you). But you need skill or they'll resist it, so you don't want to start leveling illusion late.