Bigger the speaker the bigger inertia of the cone/voice-coil assembly is.

That means one thing:  instead of stopping at the resting point , the cone assembly overshoots and thus generates unwanted  EMF ( electromagnetic force ) that produces a current that feeds back in to the audio amp.

And that’s a cause of distortion. So, what you need is a audio-amp with a very low impedance of the output stage  that will shunt the EMF generated by the speaker coil and reduce unwanted current.

Damping Factor represents the ability of an audio amp to  control unwanted speaker cone movements and is a ratio of the  speaker impedance  and amplifier output impedance  plus lead wire resistance.

Damping factor  = Speaker Coil Impedance / Amp + Wire Impedance

The lower denominator is ( amplifier plus wire impedance )  results in higher Damping Factor  and thus lower distortions caused by this aspect of speaker dynamic ( there are more distortions in an audio system ).

A good quality audio amp has a Damping Factor in the hundredths  ( over 300 ).