Riight.... I'm going to have to disagree with most everyone on this..
If you are miserable at what you do (job or school) - that is a pretty big frickin sign. The first sign I look for if I need a career change is, "Do I look forward to the weekends?" If I wish away 5 out of every 7 days (or in your case wish away four years) it isn't worth it. What's the point? If you are still interested in serving your country, check out other options (ROTC, OCS, etc.) I had a blast in college and can proudly say I had no regrets. I'm hard pressed to find a buddy of mine from the acadamy that can say that with the same enthusiasm.
One thing I've found, and not to lump everyone together, is that sometimes academy grads have a hard time adjusting to life outside the academy. You see, they, like most of us, were coddled by their parents, then they were coddled by the USNA, and then we are all coddled by the military. There was no chance to be your own person, make mistakes, and learn from it. I see academy grads (officers) acting like I did as an 18 year old college freshman.
My point here is this - yes, suck it up, be miserable or go somewhere else and be happy. What will your regrets be in life? If you are thinking about the quality of officer you will be if you don't stay at the academy compared to if you do - that is up to you and your own experiences. There are good and bad officers from all schools.