Life After SWO
As
@RUFiO181 alluded to, there's a lot of variety in post-SWO life. Grad school (mostly B-School, some law school), consulting, and GS work/gov contracting, account for the majority of follow-on options I've seen JO's pursuing in DC. Many grad schools recruit a quota of military JOs and you can get incentives on top of the GI Bill, making it a really good deal to knock out a masters. Same goes for companies recruiting JOs into middle management, GE's JOLP program for instance.
In terms of federal departments and agencies: if you can name it, I pretty much guarantee there are former SWOs working there. The FERS retirement program lets you buy back your military time, the GS world is very reservist-friendly, the DC JOPA is pretty good at gaming the USAJobs site, and being stationed in DC gives you a lot of access to network with people in the offices/jobs you want to work in. So yeah, shore duty in DC is very conducive to becoming a lifelong bureaucrat.
Working in the Pentagon
There are many SWO LTs running around the puzzle palace. They can be broken into two groups: aides and action officers (AOs).
Aides are basically 'the Admiral's bitch.' Seriously though, they run his personal schedule, take care of compiling his notes, briefs, read aheads, etc; kind of like a personal secretary. It can be a pretty career enhancing job (if there is such a thing for a SWO on shore duty) and you learn a lot because they typically get to sit in on very high level stuff that the JOPA proletariat isn't usually privy to. All that said, if you are planning to get out you don't want to be an aide.
AO/ROs work in a shop running a portfolio for their boss. OPNAV N96, as
@RUFiO181 mentioned, is the SWO Mecca of the Pentagon, though certainly not where all SWOs work. An N96 LT may own an account, say, DDG modernization or Standard Missiles. He talks to the program offices and stakeholders, assesses need, costs, and (in theory) helps the N96 Admiral and staff budget for that portfolio. Outside of 96, there are SWO LTs running around working as AOs/ROs in many OPNAV branches. Pros and cons to everything; some very boring jobs where you work great hours and can telecommute once a week; and very interesting jobs- the daily OPS briefers for CNO, VCNO, SECNAV, and the 3*s are all LTs- cool job, but the hours absolutely suck.
For your last question, you won't see the inside of the Pentagon until you're done with your DIVO tours.