Polymer retention from the flow of a polymer solution through porous media results in substantial decrease of the permeability; however, the underlying physics of this effect is unknown. While the polymer retention leads to a decrease in pore volume, here we show that this cannot cause the full reduction in permeability. Instead, to determine the origin of this anomalous decrease in permeability, we use confocal microscopy to measure the pore-level velocities in an index-matched model porous medium. We show that they exhibit an exponential distribution and, upon polymer retention, this distribution is broadened yet retains the same exponential form. Surprisingly, the velocity distributions are scaled by the inverse square root of the permeabilities. We combine experiment and simulation to show these changes result from diversion of flow in the random porous-medium network rather than reduction in pore volume upon polymer retention.