But the most successful ones (artistically speaking) are often where there isn't necessarily a narrative, but there's something conceptually coherent about the songs on an album, with how they relate to each other and what they're about creating an over-arching vibe. Best example here has got to be dear old Space Ritual, with Warrior on the Edge of Time coming a close second. The recent album from Amebix, Sonic Mass, is another good example (it's also totally super). I've always had Bowie's best album The Man Who Sold The World in the same bracket too - like Aqualung, sounds like it should be a concept album even if it isn't.
Agree on all points. Albums that tell a story always seem to be a little forced, shoe-horning clumsy lyrics into songs. Also create songs that are specific to a story, lacking a universality. If your having Man Who Sold The World then I am making a case for Queen II.