YES! I knew somebody would figure it out. As soon as I understood what “Scale Space” was, I wondered about “Scale Time” and it is indeed a possible thing but I was looking around for SOMEBODY to figure out how to implement it.
And this is what it should look like and what it does look like. I doubt this would work in real-time but for video where you have access to the future as well as the past, this is just what I’d hoped to see.
Optimizing Video Object Detection via a Scale-Time Lattice
Kai Chen Jiaqi Wang Shuo Yang Xingcheng Zhang Yuanjun Xiong Chen Change Loy Dahua Lin, 2018