It's the risk vs. reward thing again. The stones can be re-erected for comparatively little risk and effort. If the ditch could be re-dug with similarly little risk, then I'd be all for it. But it can't, so that's an easy line to draw. I'm not sure I'd have an issue with pulling the houses down though, so long as they leave the pub ;)