The loudest supporters of Brexit are on the right, but plenty on the left want to leave, too