We use both. If a quote has an offensive rating of 8 but only had three votes we won't remove it just yet. If it gets to 10 votes or more with an average of 7 or higher it will be removed. This way we are basing it on at least some form of minimal consensus.
So it is just as important to rate inoffensive quotes as a 1 as it is to rate offensive quotes as a ten.