Like any productive person, Steve Harvey would like to be left alone