I had a look at some old maps of the tor, and there's 'St Bellarmin's Chapel (Remains of)' marked... On the 1:2500 for 1882 it's a definite rectangularish wall surrounding a little building. The latest edition 1:10,000 still has it marked, but there's only three walls.
So presumably it's still there.. better get your anorak on?