It’s true. I have something similar with sounds/music (every sound is music to me) and taste but my color senses aren’t like those are. With taste, I get colors and textures and with music/sound I get almost mathematical interactions and interacting flows. Not sure how else to describe it. But with colors I don’t get those things.
The only strange thing I get with color is that I can sometimes see various colors “flashing” when I look at something. Component colors of a kind, like a math thing. “this color + this color equals that color” and they flash rapidly. So a green might have yellow blue and a light gray flashing quickly, almost like those old color wheels that spin to make new colors but malfunctioning.
That’d happen a LOT to me as a kid but I’ve learned to ignore it most of the time now except when I’m tired and then I notice it again.
Or like right now, I’m looking at a red mug near me and for some reason it’s flashing orange and yellow (and I see the red that it is inbetween as well). I guess it’s orange minus yellow? I dunno. I have no idea why my brain does that (or maybe it’s a malfunction in my retinas) but I just accept it as one of those weird things that we all have something of.