The word might be useful if you live in some rigidly conformist society where all those people can "safely" be lumped together because any deviation from the norm is to be shunned as completely unacceptable. But I sure wouldn't wanna live in such a dreary culture. My friends are "freaks" of various types, but the people I avoid are "freaks" of other types. The conventions I go to are veritable "freak-a-thons", but at a con, one may hear comments about "that freak" who's bothering everybody.
It's an extremely overloaded word.