My brother Pete has a mantra, picked up from a former employer: "If you're working hard, you're doing it wrong." The idea being that there's always some trick of the trade, some better way to arrange materials, something, to make the manual labor relatively easy.
Absolutely we've lost many techniques that made pyramids and cyclopean masonry and so forth much less physical effort than we imagine was put into them.