Your right, it's not necessary that they would break to simular sizes. It would depend on the forces that were applied to them.. A striking blow may very well shatter the stone. But a large flat stone may crack it's 'back' with the movement of soil below it.
With so much stone removed from the Downs, what is left may not be diagnostic..