Crowdsourced Ad Reviews – How Human Input Helps Refine AI Systems

Daniel Faggella

Daniel Faggella is Head of Research at Emerj. Called upon by the United Nations, World Bank, INTERPOL, and leading enterprises, Daniel is a globally sought-after expert on the competitive strategy implications of AI for business and government leaders.

Crowdsourced Ad Reviews - How Human Input Helps Refine AI Systems

For large eCommerce, media and travel companies, running huge campaigns requires substantial effort and human oversight. Translating and laying out ads for dozens of languages is a challenge that one day machines may master — but they’ll do so with the help of constant human guiding and adjustments.

I spoke with Charly Walther, VP of Product and Growth at, to learn how crowdsourced human effort is used to review and adjust ads, and to potentially train algorithms to create better ad variations.

Ad Translation

Walther talks about the ad review work being done for global online travel firm — one of’s clients:

“Expedia runs ads, targeting people all over the world. If they want to run ads for certain hotels, they want to make sure that an ad for a hotel in Brazil and China and Japan… first of all, they have to translate the ad — and they have to make sure that the layout and style all work well together.”

More than literal translation, companies must also be mindful of the marketing message behind the ad itself, and make sure that the translated message will resonate well with the native audience in that country.

“There is something going on here where people from different countries might have slightly different versions in their head of why they want to visit a certain place.”

In other words, the citizens of some countries might want to visit Paris to see the Eiffel Tower, or the Louvre, while citizens from other countries might be more interested in the cuisine. Walther gave us a hypothetical example to demonstrate the point:

“As a hypothetical example, maybe for people in China, a large beer festival might be an appealing reason to visit Germany, but in Japan, maybe it would be the little cutesy houses in the countryside.”

In addition, some terms just don’t carry over the same spark and pizzaz when translated to another language. For example, a literal translation of: “Tickets are going fast” — when translated to Japanese or Swedish — may not have the same kind of air of urgency as it did in English.

There may be an entirely new idiom used in Swedish or Japanese that should be used instead, and the implied scarcity of an ad is unlikely to be translated well without human input.

With these nuances added, however, a machine learning system might be able to train on more and more relevant copy – and a more robust advertising translation model between those two languages (say, English and German) could be achieved.

Gengo Ad Review Slide 2
Identifying grammar errors can be particularly challenging for machines to detect, and may require human filtering. Source:
Gengo Ad Review Slide 1
The meaning of a word or sentence involves context. Translated terms in advertisements or other content may require human labeling. Source:

Ad Layout

A well-translated ad isn’t necessarily a good ad. Ads that read well but don’t fit within the advertising space won’t work very well (having critical words from an ad “cut out” isn’t going to help with conversion rates). Walther elaborates:

“Literally just the length of words in an ad is critically important. A nice, one-line slogan in English might become much shorter in Chinese characters, or it might become far too long once translated into German.”

A hypothetical ad in English might show up as follows:

Bellagio Hotel – New Year’s Day Special

One of the most elegant hotels in Vegas,

Stay in early January for a full 15% off.

Might translate directly to Spanish as:

Bellagio Hotel – Especial de Año Nuevo

Uno de los hoteles más elegantes de Vegas,

Quédate a principios de enero con un 15% de descuento.

In this case, the word “descuento” (discount) might be cut out of the advertisement when it is actually displayed. A trained human would likely be able to find this discrepancy and ensure that some kind of re-wording was done to make sure the layout fits.

Cutting off elements of text isn’t the only layout concern to be mindful of. If a sentence must be broken into two lines, there is often a natural way to break the sentence – which might be difficult for a machine to determine.

Example of a sentence that is not broken well:

Sometimes a vacation on the beach is not

enough, save on the best events in Hawaii

Ending a line with “not” might be confusing, and may not read well. While a machine may not pick this up, a human being parsing the ad with their own two eyes might quickly be able to find a better way to word this ad.

Similarly, sometimes line breaks can lead to a few lingering words in a line. As a hypothetical example:

Sometimes a vacation on the beach is not enough, save on the best events

in Hawaii

The words “in Hawaii” below the rest of the sentence is strange, and takes away from the meaning of the ad. Again, this could be an apt job for human discernment. Layout involves a good deal of “feel” for wording and for word placement, and that’s the kind of work that could call for crowdsourcing.

We spoke with Dr. Charles Martin to gather additional context on the role of training data for ad reviews. Martin worked with content giant Demand Media during their boom in the last 2000s, and now consults with startups and global brands on AI projects.

Martin emphasized the importance of implicit feedback for ad review and design. The data collected from purchase behavior and clicks is more relevant than someone’s opinion about a given ad. Because of the prevalence of bot traffic, the marketers in charge of designing advertisements have to be careful to build their ad designs and “winning” ads based on financial performance, as opposed to impressions and clicks alone.

He also mentions the limits of machine moderation and translating, emphasizing that “language is fluid,” and that words and meanings often only make sense in context, context that human beings can understand. The same sentence on an advertisement for new furniture may have completely different connotations if the same sentence were read in a crime novel. Advertising copy needs to evoke the right emotions and connotations – and that’s a job that machines can’t currently do on their own – though they’re working on it.

Ad Images

Images are also a critical part of a successful ad. On channels like Instagram or Facebook, images might be much more important than the text for the advertisement.

Walther explains where humans come into the loop for the image review process:

“If you are running ads for hotels, you might have hundreds of pictures of a specific hotel that could be rotated in for use in ad variations. Occasionally there might be images that — for whatever reason — are grainy or from a poor angle, or might only feature a picture of the front desk (not the outside of the hotel itself), and a human would be able to pick that up before the ad goes live.”

While machines have made massive leaps in understanding images, real human beings might still serve an important role in determining which pictures are most appealing, or most accurately represent the product being sold.

Continuous human discernment (and feedback form the success or failure of thousands and thousands of ads for a specific product) could help machines choose images that are more likely to drive conversions.

“So, anything that might need human review to prevent accidents – this is a potential application for crowdsourced work. With things like ads, you can do a lot with A/B testing, but when they go through a lot of data, they want to make sure that nothing weird gets there in the first place.”

In addition, images can be scored manually by human beings for various criterion such as “beauty” or “fun” or “excitement” or any other abstract concept. A company with this kind of rich data attached to its images could potentially run ads just related to one of these concepts.

For example, an advertisement appealing to families with young children might be adjusted to only run ads with images that scored high for “fun”. This information could also be fed to a machine vision system which could be trained to better predict “beautiful” or “exciting” images, which it could use for future testing.

Walther mentions that this kind of manual human “scoring” is also helpful for companies because it helps to reduce the need for additional A/B testing. A company offering hotel rooms in London might only have a very small campaign running in Mumbai, India, for this particular offer. Rather than testing those ads with hundreds of images, a machine would be able to select only the images with the highest score, and simply test a few dozen which might be most likely to garner a conversion.

How Crowdsourcing Ad Reviews Works

Walther explains that ads created by human beings or by a translation engine are sent out to native speakers of that language for a round of quality checks and adjustments – ensuring that translation, layout, and marketing message are clear. Even ads created by humans (say, translating from English to Japanese) often require a native Japanese person to ensure the best wording and messaging – and Walther says that this is part of what Gengo does.

These finalized ads can then be run live by the client company, or can be used to train a translation engine to more accurately recreate specific ad types for Google, Facebook, or other channels.

About offers high-quality, multilingual, crowdsourced data services for training machine learning models. The firm boasts tens of thousands of crowdsourced workers around the globe, servicing the likes of technology giants such as Expedia, Facebook, Amazon, and more.


This article was sponsored by, and was written, edited and published in alignment with our transparent Emerj sponsored content guidelines. Learn more about reaching our AI-focused executive audience on our Emerj advertising page.

Header image credit: Truthout

Stay Ahead of the AI Curve

Discover the critical AI trends and applications that separate winners from losers in the future of business.

Sign up for the 'AI Advantage' newsletter: