For most Ghanaians, there was never any boom