After discovering the relevant anomalies I chose to start the mission to build the Baratak grove and  Manti Birthing Pools.  I initially thought that this would be an indestructible improvement since the citizens traveled to my worlds because I built them.  To my surprise I was able to destroy the improvement and the citizens stayed!

Now, as much as I love this exploit it seems as though I should have to suffer at least X number of turns of indestructible improvements.  Or if I destroy the improvements the citizens will leave.  Just a thought.