4% = a bit more than 1/32nd
From a pure statistical viewpoint, 3.125% of your DNA is inherited from your great, great, great, grandparents from 5 generations ago.
So for 4% of our DNA to be from Neanderthal origins, that would have to be in very stable locations on the genome.