Yup. They know too much about you.
Family’s a good example. No matter how successful you are, they remember when you threw up on the couch. Your schoolmates remember when you went through that “hair” phase. Your neighbors remember when you got drunk and weaved through the streets cursing at the sky.
You have to find your followers from afar. You can construct your image more easily with them.