Three of six potential host cities have now pulled out of the chance to host the 2024 Olympics