Sensibly, I guess the extent of the 'brand taint' you experience may depend a lot upon just how strongly you feel about the music and just how strongly you feel about your difference in outlook to that of the artist in question.
I can't see how listening to a record by The Smiths is going to propagate racism, any more than listening to a late '80's Nick Cave and the Bad Seeds album would give you a hankering for heroin or enjoying the eponymous Echo and the Bunnymen album would suddenly see you taking a separate private jet to the rest of the band.