Anecdotal 'evidence' says that many of the cairns and mounds that were 'robbed' in Ireland were done so during the famine – desperate times = desperate measures, hence a treasure hunt where there was little evidence that anything might be found.
Also, the road/wall building material theory is an actual fact, one that I reckon continues today.