Can I refund a perk point?

While leveling in Skyrim I put a perk in a tree, but now I feel like it is wasted, since I've found blowing things up with spells is much more fun th...