Visual Search is a feature on the Amazon App driven by computer vision technologies. I directed the redesign of the app to include a region of interest, tags, redesigned of alerts, and photo upload. This feature has been launched into the Amazon mobile app for iOS.
Originally product management had planned incremental revisions to our camera search feature for 2016. Looking across our competition and our usage stats I proposed a whole scale design revision of our app instead. Some improvements existed in the product management road map, but were planned to be executed singly and without a coherent design strategy. If we had done this, the app would have remained built around an interface design strategy that was outdated and didn't reflect our understanding of our customer and advancements in the underlying computer vision technology.
My plan was accepted and after 9 months of design and development, the newly redesigned app went live in October of 2016.
During the actual redesign my role was to direct the overall design, support the designer as he developed interaction design solutions, coordinate with engineering, scientists, front end developers and the design teams in Seattle who were impacted by our work.
We tested several approaches to starting the app. The tests involved 3 days of testing in a lab with 12 subjects. Two performed very well. This approach shows a snap approach where a photo is taken after which the user is taken to results.
Initial results begin to skeleton in the new design.
Because we used machine learning to drive the search results we did not have any text to support our results. As a result, the engineering team worked with us to create tags by referencing text searches that delivered in similar search results. This text became the platform for tagging.
In computer vision, accuracy of results depends on calling on several services. machine learning, VSS, category, text recognition, etc. all could drive results. Each set of results engages the tags differently. Delivering a consistently designed UX to the customer despite different kinds of results, is a hidden challenge of working with computer vision.