You're following. The argument is that when flying a radial, you can measure how far off that radial (or GPS course line) that you are (ie, XTK) by the deflection of a CDI (or GPS CDI/XTK distance). But when you moving from one point in space to another point in space with just a VOR/DME or TACAN, you're not able to measure any deviation from the initial required course because there is no way to depict the course line. Essentially, as the argument goes, you're homing, not navigating.
Like many things that are different between the services, I think it's a matter of comfort level. Apparently the FAA is no longer as comfortable with it either.
14 CFR 91.177 - Minimum altitudes for IFR operations. (ii) In any other case, an altitude of
1,000 feet above the highest obstacle within a horizontal distance of
4 nautical miles from the course to be flown...
You can, and probably have, looked up the rest of the stipulations.
When flying point-to-point if you can't maintain some degree of accuracy (cross track error), then you can't be guaranteed obstacle clearance or aircraft separation requirements have been met. For a jet at altitude, "within a few miles" might be fine. For a helicopter already flying at minimum altitudes, 1/2 mile of deviation might get you too close to an obstacle. Do VFR flight following near some tall towers and see if ATC doesn't require you to fly around them (ie vector you) unless you have visual contact.
Tbl 1-2-1 in the AIM shows required navigational performance (RNP) required for different phases of flight. Enroute is 2 NM from centerline. Terminal is 1 NM and approach is .3 to 1 NM. Also from the AIM, "For an aircraft to meet the requirements of RNAV, a specified RNAV accuracy must be met 95 percent of the flight time." (Sorry for the AIM vs FAR reference. I don't have my tabbed out and highlight FAR/AIM with me. The AIM is easier to search.)
You can see that poor execution of TACAN point to point probably does not fulfill RNP requirements, especially in the terminal and approach arena.