I see your point, but disagree with it. Mainly because I have found the things and know that if I wasn't looking for them I would never have known they were there.
Mostly it's just a bit of fun that harms no one and often the trash I remove on the hunt is significant. So rather than making an impact I am perhaps making a positive impact (at least it evens it out a bit).