“The earliest known version of the story in its “turtle” form appeared in 1854, in a transcript of remarks by preacher Joseph Frederick Berg addressed to Joseph Barker:
My opponent’s reasoning reminds me of the heathen, who, being asked on what the world stood, replied, “On a tortoise.” But on what does the tortoise stand? “On another tortoise.” With Mr. Barker, too, there are tortoises all the way down. (Vehement and vociferous applause.)
— “Second Evening: Remarks of Rev. Dr. Berg”
It originally came from a preacher making fun of the heathen.
But like all stories that are polemical, it changes form and it got flipped around to support the oppositions arguments.