If I was a woman I’d probably be concerned if the co-worker/boss was a slime bucket. But as a guy I wouldn’t assume more. It’s the diff between professional and unprofessional work environments. Corp cultures all differ.
If he was a known problem person, sure. Gossip channels can give it away and usually the guy himself because slime isn’t hard to spot.