If it is online, they could send private messages to each other. Or they could be members of a group that you are not in. If it is in person, they can simply make a look at each other that you don’t understand that means, “What an idiot!” that they understand.
It happens. Also, in public groups, someone can talk “behind your back” publicly by having a kind of ‘code” where they are making fun of someone in a way that the person doesn’t understand, but they do.
The secret is to understand all of the codes, but not let them know that you know the codes. That gives you insight.