Suuuure... It'd probably be easier to switch from SEAL to Navy pilot too. I'm sure NSW is ALL about it!
Actually, I cruised with a Hornet pilot in CAG 17 (don't remember what squadron) who had been an enlisted guy in the Teams before he got his comission. Kinda weird seeing somebody wearing both a trident and wings. Good dude, though.
So it can be done - the thing is, both fields (aviation and specwar/specops) require a lot of very hard work, and it's not often someone's willing to give up on everything you've done to belong to one to start all over with the other.