Only half of workers asked back to the office full-time are actually going