AI is finding itself in the physical retail space by way of augmented reality, or AR. Research and Markets estimated the size of the AR in retail market around $1.155 billion in 2018. As of now, numerous companies claim to provide AR support to shopper experiences in a variety of industries, including automotive and fashion.
We researched the use of AR in the retail space to better understand where AI comes into play in the industry and to answer the following questions:
- What types of AR applications are currently in use in retail?
- What tangible results has AR driven in retail?
- Are there any common trends among these innovation efforts? How could these trends affect the future of retail?
This article intends to provide business leaders in the retail space with an idea of what they can currently expect from Ai in their industry. We hope that this article allows business leaders in retail to garner insights they can confidently relay to their executive teams so they can make informed decisions when thinking about AI adoption. At the very least, this article intends to act as a method of reducing the time business leaders in retail spend researching AI companies with whom they may (or may not) be interested in working.
In this report, we explore the uses of AR combined with artificial intelligence in the following spaces:
- General Retail
- Home Decorating and Improvement
- Clothing and Apparel
- Automotive
General Retail
Aisle411
One company that claims it is developing AR technology for retail supermarkets is Aisle411. The company created a shopping application for Walgreens which integrates AI into AR technology to help shoppers find products more efficiently. The application, which runs on a tablet attached to a user’s shopping cart, makes use of Google Tango’s computer vision-driven 3D-mapping service. Google Tango is used to guide in-store shoppers using a downloaded map.
Customers can also search product names, item numbers, and product categories by voice and command the mapping app to give them store navigation instructions to a product’s exact location, as shown in this 2-minute video below:
According to the company, Google Tango determines the shopper’s position and orientation inside the store with a precision of up to a few centimeters, while the tablet displays products offered by aisle.
Our research led us to a report saying the app was free for download from the Apple Store, but the page would not load. On the Google Play store, the app received an average 2.6 rating from 24 reviewers.
Aisle411 also developed a similar application for Toys”R”Us, prior to its 2018 close. This application was deployed to the store’s 800 outlets and required shoppers to scan a QR code at the store entrance using smartphones. While details were not given about how long this service was implemented, the case study stated that users of Aisle411’s app had “34% higher tickets,” which seems to mean a large purchase amount, than the average Toys”R”Us shopper.
Aisle411 also lists an unnamed Las Vegas casino and the San Jose Internation airport as clients.
Niarcas Jeffrey is CTO at Aisle411 and an engineering lead at 1904 labs. He earned his MS in Computer Science from Washington University in St. Louis.
Home Decoratingand Improvement
IKEA
IKEA offers IKEA Place, a mobile app that purports to allow customers to see how a piece of furniture would look and fit in a given space.
Launched in autumn 2017, the mobile app is built on Apple’s ARKit platform that supports 2D and 3D image recognition and image tracking. This technology could allow users to move still photos of furniture into a still image of a room.
After the user takes a photo of a room on the app, IKEA states the AR capability is able to measure the dimensions of the room and allow the user to drop an image of a piece of furniture into the photo from IKEA’s 2,200-item curated catalog. According to IKEA’s website, the application is accurate up to a millimeter.
The short video review below shows how IKEA Place claims to virtually place a piece of furniture in a room:
The app received a 4.7 score in the Apple Store from 1,800 ratings. Sensor Tower, an app analytics platform, reported in March 2018 that IKEA Place is the second-most downloaded ARKit application from the Apple Store. IKEA has not revealed any revenue figures that have resulted from the app.
IKEA contracted Twnkls to develop the application. Lex van der Sluijs is CTO at Twnkls, although we did not find evidence of a robust AI background anywhere among Twnkls team.
Wayfair
Online furniture retailer Wayfair also offers an AR app featuring its 3D library. According to Wayfair, the app is built on both Apple’s ARKit and Google’s ARCore platforms, giving developers a way to integrate AR technology into both iOS and Android applications.
On the Google platform, Wayfair’s app uses Tango’s mapping, computer vision, depth-sensing, 3D-motion tracking, and machine learning technology to allow customers to view on their Android phone or tablet how a piece of furniture would look in their home.
Similarly to IKEA Place, shoppers first load a photo of a room into the software. After browsing Wayfair’s product catalog and tapping on their preferred piece of furniture or décor, the user then taps the “View in Room 3D” button under the product image. The image of the product then appears in the photo, according to Wayfair.
Shoppers can move products on the screen and physically walk around to view different angles, and they can purchase the item within the app. The company claims that all 3D products are sized to true dimensions so that shoppers do not need to physically measure an item or their space beforehand to know that it will fit before they purchase. We assume that the AI comes into play during this process,
Wayfair claims the app uses computer vision to allow customers to take photos of items they like in real life and find visually similar products on the app and present users with pieces of furniture thye might like based on their past use of the app
The 1-minute video shows how users can view pieces of furniture such as sofas, chairs, and lighting fixtures using the app:
Dan Wulin, Director of Data Science at Wayfair, received a PhD in theoretical physics from the University of Chicago and earned double Bachelor’s degrees in mathematics and physics at Columbia University. Prior to Wayfair, he worked in strategy consulting at the Boston Consulting Group.
HoloLens
Microsoft’s Hololens is a mixed-reality (MR) headset that enables users to engage with digital content and interact with holograms. Microsoft claims Hololens has uses in a variety of industries, including home decorating and improvement. The company claims the software is equipped with:
- Sensors that follow the user’s eye and head movements and a cursor that allows users to select holographic images.
- Gesture recognition, allowing users to move their bodies in order to open apps, select and size items, and drag and drop holograms in the augmented world.
- The Cortana speech recognition-driven virtual assistant, which allows users to navigate, select, open, command, and control the app by voice.
HoloLens has a home improvement use case. Lowe’s Innovation Labs wanted to help better visualize kitchen renovations, rather than just offer samples of paints. It partnered with Microsoft to pilot Lowe’s Hologram Experience using HoloLens in two stores. According to Microsoft, the resulting application from the partnership enabled shoppers to view a mix of physical and holographic objects merged through the HoloLens headset.
Shoppers can start using the application once they wear the headset in the Lowe’s kitchen showroom where the AR experience is staged. A human HoloLens specialist is on standby to guide the shoppers through the “renovation.”
Using speech, shoppers can activate Cortana who will show them the holographic images of kitchen appliances or fitting. Using pointing or flicking gestures, shoppers can choose which products will be placed in specific places in the staging area.
For example, they can ask the application to change the color of countertops, the tiling of backsplashes or even the cabinet styles. The app will also display hardware and appliances.
The 7-minute demo, in partnership with Lowe’s, shows how the Cortana virtual assistant’s deep learning algorithms were trained on millions of images to find design patterns and make recommendations to customers.
HoloLens is also used by the Case Western Reserve University, University College London, Ford, and the Bedfordshire Police. The company also works with partners such as Avanade, Kognitiv Spark, Sapient Razorfish to bring the technology to their own clients.
Harry Shum is Microsoft’s Executive Vice President, Artificial Intelligence and Research. He holds a doctorate degree in computer science from the Carnegie Mellon University. He joined Microsoft in 2004 as part of the research team. Over the years, his work has encompassed computer vision and graphics research, establishing the Microsoft Research Asia lab, leading engineering for Bing, and heading Microsoft Research globally.
Clothing and Apparel
Warby Parker
Online eyeglass retailer Warby Parker offers the Glasses mobile application that allows customers to fit eyeglass and sunglass frames online.
Warby Parker’s site has a try-on feature that enables customers to upload a photo of themselves or capture a photo using their laptop’s webcam. When taking and selecting a facial image, a user is prompted to match up their head and ears with white dotted lines on the left and right of the screen.
If the customer likes the webcam’s snapshot, they can continue and submit a valid eyeglass prescription and pupillary distance (PD). The PD is the distance in millimeters between the eyes’ pupils, and it can be measured using a tool in the app. The PD is important for centering the prescription correctly in the frames.
Once the requirements are submitted, the customer can try on any pair of glasses from the collection and make slight adjustments to where the frames rest.
Once the image is uploaded, the facial recognition technology measures and maps the customer’s face to determine which frame shape would be the best, and a recommendation engine suggests some models. As the customer clicks an eyeglass model, the AR technology layers the glasses onto the customer’s face.
We were unable to find a demonstration video for the app.
After scanning and calculating the results, the app’s algorithms go through the catalog to match the customer with eyeglass models that would best fit the customer’s face.
We were unable to find any C-level executives with AI experience on the company’s team.
Holition
Holition is a digital creative studio that offers Face, which the company claims allows users to try different makeup looks via an in-store virtual mirror or mobile app using computer vision.
The application can be installed on iOS and Android smartphones and tablets. When the user opens the app, it uses facial-recognition technology to scan the user’s face so that it appears on the screen ready for augmentation. Users can then click on the makeup product they wish to apply to their virtual face, such as lipstick, eyeshadow, blush, and foundation. At the bottom of the screen is a color palette from which the customer can tap to apply the preferred shade on their face.
The 2-minute demo below explains how the technology works:
Charlotte Tilbury asked Holition to develop two interactive magic mirrors for the launch its flagship store in London’s Westfield Shepherds Bush. As shown in the video below, each touchscreen Magic Mirror lets customers try 10 makeup looks.
The app incorporates skin tone detection and its recommendation engine suggests the best look for the customer, according to Holition.
Holition’s team worked with the client’s makeup artists to learn how they apply makeup in-store. This included the makeup artists’ strategies for layering and blending. The process was replicated digitally to create a realistic visualization, according to Holition. Holition also notes that one magic mirror feature allows users to go shuffle through 10 looks in 40 seconds as it takes a snapshot and emails their preferred choice.
Holition claims that its other clients include The North Face, De Beers, and Tag Heuer.
Russell Freeman is the CTO at Holition and has been with the company since 2011. He earned both a PhD in Augmented Reality and a Master’s degree in Vision, Imaging and Virtual Environments from the University College of London. Prior to Holition, he worked at Philips, which makes sensor products for industries including healthcare.
Automotive
Blippar and Jaguar
Jaguar Land Rover says it has employed AR technology to allow prospective buyers to explore the latest SUV. The AR feature, developed by UK-based Blippar, appears after an online user clicks on a banner ad on either Android or iOS phones, as well as Chrome and Firefox browsers.
According to Blippar’s case study of Jaguar, the AR experience launches when the user clicks on a banner advertisement, a technology it calls augmented reality digital placement (ARDP), a web-based AR. Blippar claims that ARDP can be run with any size of ad placement on any network or publisher and that it works with any desktop browser, most mobile browsers and device combinations.
As a result, the company reports that users need not download an application from the Google or Apple store. As seen in the video below, tapping the banner advertisement gives the mobile-optimized site permission to use the smartphone’s camera:
Blippar claims to use AR and computer vision to put prospective customers in the virtual driver’s seat, where they can potentially explore the car interiors or through the windows.
Blippar claims its computer vision-enabled car recognition application programming interface can also be integrated into mobile applications by developers from other car companies. This information can also be integrated into Blippars app, which allows users to scan nearby or photographed vehicles and see content relevant to that make and model. Depending on the content source, the app map will display dealerships, insurance quotes, pricing and car specs, and allow a user to book a test drive, according to Blippar.
Blippar has created similar features for Porsche and offers slightly-modified AR experiences for non-automotive clients including McDonald’s, Coke, and Spotify.
Luca Del Pero is Head of Computer Vision at Blippar. He earned his PhD in computer science from the University of Arizona.
Concluding Thoughts
Combining AR and AI in business may provide a marketing tool that can also gather data to help companies learn more about shopper preferences.
While the technology is still relatively new, all of the above retailers claim to have implemented it because it gives the customer:
- A view of digital representations of products before they commit to purchasing
- Information about what they’re buying
- The possibility of seeing options not available in the physical location
- Recommendations based on their preferences
Currently, while case studies were not available, revenue and app downloads associated with IKEA and Wayfair show that AR for placing furniture may become more of a common feature among other furniture and home-improvement retailers. While the Lowe’s experience is more in-store friendly, it still follows a similar trend of using computer vision, staging measurements, and AR technology to allow users to preview furniture, kitchen upgrades, and other home additions.
Header Image Credit: MIT Technology Review
