If it makes you feel better there is no crime of trespassing in England. The land owner can ask you to leave the property but that is all. If they do anything more then you can report them to the police should you wish to for threatening behaviour.
The only thing you could get into trouble for is if they can prove actual criminal damage, which is unlikely.