Dan Hoffman takes a job as a “search engine evaluator”—a Sisyphean nightmare for $13 an hour.


In the winter of 2011, my roommate, whom I owed money, got me a job as a “search engine evaluator” for a mysterious organization known as Leapforce. Actually, he was farming out the work to me: he would keep his contract and pay me a percentage of the $13/hour rate while I worked under his name. All I needed to do was read over the various manuals they had given him, and I’d be set.

Learning the ins and outs of the job ended up being about a day’s worth of reading. My job would be to evaluate search engine results based on their relevancy to a given “query,” a query being what someone might type in the search bar on Google, Yahoo!, or Bing.

There were several different types of “tasks,” but most often I was assigned “URLs” or “SxSs” (side by sides). For the first, I was given a query, which ranged from things like “ipods” to “how to pop pimples” to “hot teens,” followed by a possible search result for the query, the URL in question . After looking at the URL’s page for about a minute, scanning its content and examining the layout, I assigned a rating. The highest was “vital,” meaning the URL was a link to the official website (if there was one), followed by ratings of “useful,” “relevant,” “slightly relevant,” and “off-topic.” I was also required to “flag” porn and/or spam. SxS tasks were the essentially the same, except I evaluated two sets of five results at a time instead of just of one search result.

I quickly realized that what I was doing didn’t make any sense to me. I had no idea how this work was useful for Leapforce in any way. This was made starkly clear to me when I was supposed to rate results for a query I didn’t understand. For example, if the query was “new hdr techniques,” I might not know what that means to begin with. For cases like this, I was instructed to learn about the query by looking at Google search results. This was perplexing on two levels: First, this assumed that when I googled the query, these results were relevant. Second, it often revealed that the result I had to rate would never actually be a result of a real search. What relation, if any, did the sample result have with search engines?

With the “new hdr techniques” query, I figured out that it was a digital photo technique that manipulates three different exposures of an image. But the URL I was supposed to rate was for a website about film restoration — a site that was completely irrelevant and would never show up in a real set of search results. (I knew this because I had just looked it up to learn about it.) How, then, is the rating a URL that would never be a search result useful to anyone?

I never found the answer to this question, and I never really understood what Leapforce actually was, other than something that generates work with no utility at all for anyone. Even when I had worked in factories, I knew there was an actual product made or an end result, even if I never saw it and I was extremely skeptical as to their value. But with Leapforce, I was even further alienated from the labor, because there was no tangible product I created or service I provided, and it was as if search engine evaluation was a version of the “light industrial” factory work I had done transplanted to the Internet.

I became considerably more depressed the longer I worked at Leapforce. Each day I woke up, drank instant coffee, and sat at our dining room table looking at the results for things like “honda civic 93 coupe” or “how to clip your toenails.” The most awful part was the amount of attention it required, which was just enough that I could never zone out or get into a groove. I attempted to make working as pleasurable as possible by chatting on either Facebook or Gmail, listening to music, and writing emails at the same time. But I realized that I needed to concentrate more on Leapforce alone. Then, after doing work for five minutes, I realized that evaluating search results was insufferably boring and I had to get on Facebook and Gmail to make it more bearable.

I never realized just how profoundly my work could affect my feelings. There was no way in which I could feel as if I was doing anything meaningful or satisfying. Even pouring someone’s coffee, for example, would have a tangible result, and I would at least get to interact with people.

I was fired. It turns out that they evaluated my performance and found it lacking. (Judging my performance confused me all the more: if they knew already what these URLs were rated, then why did they hire people to rate them? I know it was more complicated than that, but it still made the whole thing seem more suspect ). But I never felt so good about being fired. I was all set to move back in with my parents for a month to save up money, and it didn’t even seem that bad. At least, it seemed better than doing Leapforce forty hours a week.

Leapforce was a uniquely dark period in my life. I am not a person given to serious philosophical introspection about the nature of life, and I’m more interested in neurotic, pragmatic introspection. It always seemed to me that thinking Big would just make me unhappier than I already was, so if my job kind of sucked, for example, why think about it philosophically and make it seem even worse? But it couldn’t be helped with Leapforce; it was just so pointless and unlike any job I’ve had before it forced me to think about my life in abstract terms and how a large portion of it was spent doing something completely devoid of meaning.

Dan Hoffman is a barista in Nashville, Tennessee.