A plan for the free movement of Africans across the continent is being held up by fears of xenophobia