To a point it's personal preference. However, there have been studies on this very subject. I'm reminded of one in which people were shown several pictures of human faces (some barely recognizable as such). It found that different people tended to rank the same faces with great regularity in the ordering. In other words, our recognition of human attractiveness is at least partially built in at a genetic level.
Extending the concept, it would seem to fit with the observation that most fetishes (following jer's tangent), if not all of them, are quite possible to trace back to a given causality rooted in genetically-determined psychology. This means that there's an explanation for most or all of them that starts off at a reasonable point and mutates to all hell.
Let's take a simple (or simplified) case example of the oft-commented-on Asian bondage fetish: Society encouraging repression + ever exacerbated overcowding -> deep, serious repression (requires an outlet) -> violent outlet + requisite privacy -> rape -> rape pornography. Add fear of law -> (usually) faked or tamed rape porn -> asian bondage. At least, this has been one explanation put forth to explain the disproportionate popularity of such stuff in Japan and similar.