I think the general consensus is that it doesn't matter where you come from - a dbag will still be a dbag
I'm sure there's pros and cons of networking and name for your school (Naval Academy vs. other universities out there) so if you're worried about that, weigh the schools you are going to. Some schools have bigger names and alumni base than others, and in some regions of the country, all the big college grads will come from one or two big name schools and that's networking/name in itself. USNA is definitely a big name and it would certainly be a bigger name for a civilian than a random state school. On the other hand, for instance, if you get into Stanford - a top 10 school in the nation - your school's reputation and all that will be fine (and of course, in the end after you get experience, how you perform and act will matter even more than the school you went to).
Also, having known high school friends that went to the Academy, make sure you are absolutely committed to it. You don't want to drop out after taking the spot of someone who might have wanted it more but didn't get the opportunity to do so.
Personally? If you aren't sure about going to the Academy, don't go. People go to school and a new environment and can change rapidly - hell, I would barely recognize myself from my pre-college days to when I graduated. It's great you want to join the Navy, so find out about NROTC at the schools you are applying to, and go kick butt at the school you choose. And yes, work hard, and play harder - you won't get another chance to live like an undergrad