Yes, the physicality of time is something I’ve noticed as well. It’s also not a “straight line” either or a block but connects in a myriad of ways, all hidden in the ways we talk about time, I believe.
“gap” or “bump” is about the best way I can think of it. Like if you’re a 1D line, the 2nd dimension would intersect and appear strange… a gap, a bump, something… “not quite regular”. Even if it’s just pencil marks on a page drawn in a direction you can’t perceive because you’re 1D you might ask, “Why is this different?”