The high-tech retail store by Amazon that surfaced last year opened up newer avenues for the customers, where they could swipe their smartphones, pick the items they want and check out without waiting in long queues. Amazon Go, which is the first ever convenience store by the tech giant extensively uses computer vision and deep learning algorithms to enable shoppers a user friendly experience. With a range of products, from ready-to-eat food to grocery essentials, the entire store can be accessed with an account on Amazon and Amazon Go app that can be easily installed. The trial which was earlier open only for Amazon employees, has now been opened for general public from this January.
With ambitious plans of opening 20,000+ stores in the coming days, Amazon Go is the combination of artificial intelligence, computer vision and data pulled from multiple sensors that ensures customers have a hassle free experience. Though the company has refrained from divulging detailed notes on how the technology works, the patent application filed by the company has some information revealed in bits and pieces, that we have tried to sum here.
Technology similar to self driving cars
Technologies such as computer vision, sensor fusion and deep learning that underlines Amazon Go, is largely used in self driving cars. Just the way self driven cars are loaded with sensors to make it learn how to drive, Amazon Go stores use cameras to track a user’s action in the store. This is where computer vision plays a crucial role, allowing computers to process visual information and generate appropriate algorithms based on the data that is gathered, to generate relevant results.
The patent filed by the company reveals that it would use a system of cameras, sensors and/or RFID readers to identify shoppers and the items they’ve chosen. It also suggests that from the time a buyer enters into the store to the time he checks out, cameras take photos of various actions—essentially when they enter store, while removing items and while keeping the items back in shelf. This data along with facial recognition and other user information like height, weight, user biometrics, purchase history etc., is being reportedly used to sketch the final iteration of these stores, and this is where AI play comes into picture. It is speculated that the cameras used in Amazon Go use RGB cameras, depth sensing cameras, and infrared sensors. Based on the data collected, they are trained to identify the user and activities they carry out inside the store with the items on display. According to filings, a shopper’s past purchase history may also be used to help identify an item when it is picked up.
“For example, if the inventory management system cannot determine if the picked item is a bottle of ketchup or a bottle of mustard, the inventory management system may consider past purchase history and/or what items the user has already picked from other inventory locations, and suggest that user is likely to pick ketchup bottle”, it explained. It also suggested that data from other input devices may also be used to assist in determining the identity of items picked or placed in inventory locations.
The second important part of the overall working is the Sensor Fusion, which is essentially an amalgamation of sensor data from a number of inputs, such as weight and motion sensors. When paired with computer vision, it helps in determining if someone has picked an item or placed it back. This may also be underlined with Trigger switches where inventory tracking mechanism would trigger a “item lifted” stage.
The final part comes in the form of deep learning, a part of machine learning that enables computers to learn by continuously collecting and analysing digital data. Combining these three tech advancements pulls out the best of Amazon Go. Some experts also speculate that weight sensors similar to those used in Walmart and Target, and motion monitoring systems.
Challenges on the Go
Given the scope of technologies that Amazon uses, they have the potential to learn more about each of their customers and scan their typical buying habits. With AI into picture, it highlights the learning capabilities, where based on the customer data, Amazon can tap into the rich pool of data collected and use it to their benefits such as offering tailored products to the customers, hence bringing more personalisation.
While the use of advanced technologies can detect any and everything that is picked, a system like this can be vulnerable and prone to errors. What if, Amazon’s AI gets confused between the users who are shopping in a close proximity, or those that could be charged incorrectly! The possibilities can be many, and to offer an error free functioning, tech giant has to keep working on overcoming existing obstacles by improving their tech offerings continuously.
