intelligence is slippery...
...and definitions vary.
But one definition is 'speed of learning' which includes new data and new processing--speed of programming. (Paths being ~ equivalent to programs) This ability is directly related to how often you exercise the ability--if you stop learning, after awhile you will find it much more difficult to learn.
"There seems to be a tendency among higher IQ to not train their bodies to the extent they are distracted by the need to feed their minds. "
It's a perception, but it isn't necessarily true because people see it that way. Exceptional talent often comes with multiple abilities, often including physical abilities.
It's far too late to start worrying about using chemistry to rank up the brain--we've been doing that for centuries