As with everything, different people have different priorities. I greatly value my experiences when put in a leadership position and having to figure how to tackle whatever hurdle was in front of me and hopefully, I've done it reasonably well at least half the time. I also always understood going into the Navy that the "price" to fly was to continue to do well at the ground job (and all of the many facets that actually means).
However, my end goal has never been to make it to the top. Instead it was to continue to fly, so I made choices to make that happen. Many here, I'm betting, have a similar feeling. It wasn't until I hit the 16 year mark and was willing to take non-flying orders to reach retirement that I changed my overall goal, but that was a means to an end to get a pay off for the rest of my life. The fact that my non-flying orders turned into flying orders was even better.
That said...
Then I'd argue you need to be prepared to walk. If your career isn't tracking in the direction you want after your first commitment is up, do something else. I know you know this, jtmedli, but speaking generally, I sometimes have issues with someone who says they won't fly for the bulk of their 20, but have only been in for 6-8 years and don't know what might be coming their way.
If the next set of orders isn't what you want, jump ship, but if it is, play it by ear. Don't take the money and instead set yourself up to be able to walk when you want to. Just don't forget to also play the long game.
I've done a bunch of OCS interviews and one of my questions/discussion points is to talk about how little we fly compared to the total hours in a year or tour. It helps give the applicant a frame of reference on how much ground work there is. I'm not sure it's always received, but hopefully some of them will think about it at least for a minute before heading off to OCS.