Recently, I was asked to fill out a questionnaire during check-out at a hotel in India. I was very pleased with my stay so I agreed to provide feedback. When I offer feedback, I am in one of two extreme emotions: I either love the service or, more commonly, I loathe it. There is no time to talk about the average. And I have given up on Comcast.
The form had about twenty questions asking how satisfied I was with various components of their hospitality. I had to choose between one and ten, the higher number for greater satisfaction.
I decided to set a record for the fastest completion of the questionnaire. I quickly chose 9 and 10. To appear objective, I gave a 7 to a service, randomly. Seven meant “above average”. Nine and ten meant “outstanding”—that is satisfaction cannot be measurably higher.
In the section which asked, “How can we do better?” I said “Put some more trees.” I didn’t really think the hotel premises needed more trees, but I was on a roll of objectivity. I had to say something.
I was about to leave the hotel when the manager stopped me. He looked worried. “Sir, what can we do to make the fitness center better?”
“Nothing.” I replied, puzzled.
“The fitness center and swimming pool are the pride of our hotel. We have the latest elliptical and treadmills,” he protested. I nodded in agreement, still confused.
“Sir, you gave the fitness center 7 out of 10. If we get less than 8 out of 10, I look bad,” he continued. I glanced at the form. Just above the question about the fitness center was one about in-room dining.
“Sorry, I picked the wrong one. You are right the fitness center is excellent. I meant to give in-room dining 7 out of 10. The masala chai is nice but has too much masala.”
I was lying.
I never stepped inside the fitness center. The masala chai was excellent, which is why I had six cups a day. But the manager seemed relieved that it wasn’t the fitness center.
“Sir, there is no space for more trees in the hotel.”
“But you must plant a couple of trees in the drive, or somewhere.” I wasn’t relenting on that one. My credibility depended on something about the hotel remaining pseudo-unsatisfactory.
Since my flippant remark about paucity of trees did not have a score—that is, it was qualitative rather than quantitative—the manager let it go. But he did offer a free ride to the airport, and masala chai with less masala.
I don’t know who designs multi-attribute surveys, whether they are over-intellectualizers, have any friends, or the type who fill a 30-question feedback form about an espresso from Starbucks. But these surveys have become ubiquitous. A variant, the Press Ganey, will help determine hospital quality and provider bonuses.
The multiple dimensions have an aura of scientific precision. But these surveys are, to put it gently, useless. They have as much granularity as words in our contracted vocabulary. Awesome is the new mean. Average means below average. Super nice means nice.
Thus, 7 out of 10 is bad because 9, outstanding, is the new normal, and not because people are uniformly outstanding, but because we can’t be bothered to discern. Yet some will be persuaded that a hotel which scores 7.27 is meaningfully better than one which scores 7.22.
I once was filling out such a form for a medical resident applying for a job, who listed me as a referee. There were over forty questions about various components which measured the resident’s critical thinking, work ethics and professionalism. It looked like a psychometric test that people joining the Central Intelligence Agency take.
I marked the attributes with fours and fives, the highest scores. Of course, the resident’s strengths were not uniform in all respects. But he was an affable resident, had impeccable work ethic and was a competent radiologist.
That was all that mattered, so I paid scant attention to the “he actively seeks new knowledge and applies evidence-based medicine.” Do these questions really add more signal than “he is a jolly good fellow and so say all of us?”
Though I find answering about multiple dimensions a nuisance, it does not mean such attributes do not exist. But it is a leap of faith by those writing these questions to think that responders will cognitively burden themselves by separately thinking about the various attributes.
Despite our grand attempts at objectifying subjectivity we remain subjectively objective. Not only is the pursuit of such objectivity an utter waste of time, it is faux precision. To think that all sorts of statistics are run on these scores, as if they are continuous variables such as temperature and length, reminds me of the quip attributed to Mark Twain: Lies, damned lies and statistics.
In the meantime I would like to apologize to the guests of that hotel if the masala chai now tastes odd or if they have to fight the bushes to get inside. Just remember to visit their gym.