You're definitely right that this is happening, but you also have to be careful about the definition of "small farm". That's often based on revenue categories that are infrequently adjusted for inflation.

Some of the decline is just farms who's nominal revenue rose above the threshold, even if the farm is the same size.