This question originally appeared on Quora: What’s it like to receive a question mark e-mail from Jeff Bezos at Amazon Inc.? Answer by Greg Hullender, computational linguist.
At Amazon, I lead the “Matching” team, and I would get “?” emails about once a month — generally whenever a merchant had a problem involving Amazon listing their products in the wrong part of a catalog. (E.g., “I was selling the paperback but you listed me on the hardback and then customers got mad at me!”)
Generally, the email would go from a merchant to some friend of the merchant’s who had some personal connection to Bezos. That friend would send it with some comment like, “Jeff, you should look at this.” Jeff would forward it to his senior VP, Sebastian Gunningham, adding just “?,” and Sebastian would forward it directly to me. Since it happened so often, eventually Sebastian took to sending it straight to me and not cc-ing anyone else.
I’d read it carefully and then assign it to someone on my team to review the logs and determine that the problem was in fact caused by our software. Much of the time, it would turn out that the “switcheroo” had been caused by something else. Other times, it would turn out that the mismatch had occurred long ago (one case took five years between the mismatch and the time a customer actually tried to buy the product). And sometimes we’d find a real bug.
While that research was happening, I would contact the executive escalation team, and they’d start putting together a formal response. They’d make sure we had the names of all the folks who really ought to be involved. That’s especially important if the root cause lay outside my team. If the research looked like it would need more than 48 hours, we’d put together a preliminary response — possibly just for Sebastian.
Once we had a concrete explanation, then I’d draft a formal response and run it by the executive escalation folks. They’d point out things that were hard to follow or which seemed likely to provoke a negative response. When it all looked good, we’d send it off.
Good responses would be things like:
1) The error was ours, but it was fixed long ago. If he had submitted his listing in the past 12 months, it would have been matched correctly.
2) This uncovered a bug, which we will fix in a deployment in 30 days.
3) The original match was correct but it was overridden by a customer service representative using a manual override tool. We have added a warning message to the tool for cases like this.
Executive Response would nag me every few days if the response were dragging out. It was generally better to send an incomplete response rather than take weeks and weeks trying to make a perfect one. Bias for action counts at Amazon.
Bad responses would be:
1) We don’t know what caused it; nothing in the logs is unusual. That would prompt a response asking, “So what are you doing to make sure you’ve logged the right data in the future?” “Diving deep” is another Amazon leadership principle.
2) We determined it isn’t our problem. Nope. Ownership is another virtue at Amazon. Especially since other teams were less skilled in this area, we’d drive them to get the answers. I wasn’t above threatening to tell Bezos that the root cause was “Senior manager X in group Y was uncooperative.” That got me yelled at a few times.
3) AI problems like matching have a certain error rate which we simply have to budget for. Although this is the truth, Jeff and Sebastian already knew this. There would have been no excuse for failing to learn something from this particular problem and trying to find a way to reduce our error rate.
A few times I did have to report that, despite many days of study, we were unable to identify the cause and that the best we could do was guess and add some new logging code to try to catch it if it happened again.
Because I had to deal with so many Bezos escalations, I was quite good at it, and would often help others inside my division, who would generally freak out when they got a “?” email.
Finally, I determined early on that problems with book volume numbers were the cause of over half of the escalations I got from Jeff. If you order volume 2 but get volume 1, you’ll be unhappy even if every other aspect of the book description were the same. Our project on “small but significant differences” eventually reduced the number of escalations that I received by over half.
More from Quora: