Immigrants have already started leaving the US