A diminished average sperm count covers the last 50 years. What about other segments of history? What about the factors contributing towards this, as well as, other effects on female and male reproductive systems; which have not gained media/'newsworthy' attention?
Of course, we may never know, unless science evolves processes to determine such.
I would imagine this is just another ebb or flow point in our development as a species with more contributing factors than can be accounted for.
What is it's consequence in the grand scheme of time, space, and present existence? It's safe to say humans (as we know them) have not always existed and someday will exist no longer, possibly in less than "a-blink-of-an-eye" much less a failed reproductive system.