To the OP, the odds are very much against you if you go this route. It is very easy to be a "middle of the road" SWO junior officer. Breaking out to the point of getting a BUD/s spot is quite rare.
In all fairness, it's very easy to be a middle of the road anything. There aren't a lot of SWOs (or Nukes/Pilots/NFOs) putting in for BUD/S slots. I'd be curious to see numbers of O's who sucessfully lat transfer to NSW/EOD.
I'm really just playing devil's advocate here; if you absolutely positively want to be a SEAL first and foremost, you should enlist.