Yes, for many people “not noticing” is considered a sign of rudeness. Like you said, it’s a reaction-thing more than an intent-thing… although there is an implied intent (because you did think them important enough to notice).
So they’re partially correct… in a strange way I suppose.
Still, the whole “Social Game” is annoying just the same. “Don’t notice me” = “You’re rude” = “I’m not important to you” = “You don’t love me”… etc. Very mathematical game… and I don’t care for playing it.