Happened to me on Saturday after GOD restarted the Game. With the restrt I found that I had lost the Ancestral Cooldown and was able to stay as long as I liked.
I had this happen today too. Except for me, it did encounter some sort of bug before the counter went away. Leaving the lands did not initiate a cooldown, but reentering did restart the timer. Didn't wait to see if it would kick me out a second time. Attached a picture of the bug/area.