“I really liked those shoes I just saw…”
“I want them but I don’t know what they’re called!”

Not to worry – we’ve got a solution for you! In March 2019, the Mercari marketplace app introduced its brand new image search feature.

So what can image search do?
When a user takes a photo with their camera, Mercari will search through the data of nearly one hundred million items that have been listed on the app, and display the most similar-looking items in the search results. This new feature can help users transition from the “want” phase to the “bought” phase even quicker than before!

 

Image search was actually an idea that picked up speed in AI Engineering, a team focusing on developing new features through machine learning. So why did we choose to release the feature at this time? We interviewed machine learning engineer Ryusuke Chiba (@metalunk) and product manager Yasuo Hishii (@hisshy) who worked on this project. They shared their thoughts on the difficulties of “machine learning x Mercari” and about how selective they had to be in order to preserve the user experience.

Image search: Born from Kandou listing and customer service tools

ーMercari’s image search feature has been released! I’d like to ask more about this project – when did it start?

@metalunk：I’ve been in Mercari since April 2017. At the time, the AI Engineering Team was already talking about making a search feature that can use image data.

@metalunk (Machine learning engineer)<

ーSo it started around 2 years ago?

@metalunk：That’s right. In order to create a feature involving machine learning, we had to repeatedly conduct inspections. A prime example is the Kandou listing feature, which has extremely similar logic to the image search feature. This is a feature that analyzes the image taken by a user during the listing process, and automatically provides information like item name, category, and brand. After that, we developed an image search feature allowing the Customer Service Team to find illegal item listings. From there, we realized that we had the opportunity to release it as a feature for the customers as well.

@hisshy：I joined Mercari in July 2018. I was originally interested in using image data for developing features, so right after joining I talked to Tairo Moriyama (Former Data Engineer Director). I mentioned, “wouldn’t it be great if we could search using just images?” and his response was, “actually, we’re already working on that.” (laughs) Not long afterward, I was put on the project.

@hisshy (Product manager)

ーSo you really joined at the best time possible. Did the project really get serious starting around July 2018?

@hisshy：Yes. Although we actually didn’t have enough development resources at the time. The members involved were working on it as a side project. Because of that, we worked on it little-by-little with our very limited resources. After new members joined at the start of the year, we were able to give the development a major push.

The ever-increasing difficulties behind “machine learning x Mercari.”

ーSo the image search feature was born and completed thanks to the Kandou listing features and customer service tools. Even though the idea was there from the very start, why do you think no one specifically suggested the creation of the image search feature?

@metalunk：To be frank, image search is a feature that requires a huge amount of work. Because Mercari is a CtoC service, there are difficulties involved that are quite different from those faced by EC services.

ーWhat were some of those challenges?

@metalunk：We’re fortunate that Mercari is being used by so many people. This means that many new items are listed every single day. Image search results use the images provided by users who list their items. As long as users continue to list items, the search results need to be updated.

ーSo the unique difficulty behind “machine learning x Mercari” comes from the fact that items are always being listed, and search results are always increasing?

@metalunk：That’s right. If 1 million items are listed per day, 90 million items will be available as search results after 3 months. This means that they become searchable through the image search feature as well. We used the AI Engineer SysML Team’s in-house platform to create a system that would automatically generate and regularly update specific data. Details are written in the Mercari Engineering Blog so you can check that out to see more about the technology we used.

ーI see!

@metalunk：In addition to that, the image search feature has to work together with components that increase the machine learning accuracy, such as object recognition and feature extraction, while making sure everything operates properly. The development process felt like we were building a pythagora switch.

ーA pythagora switch?

@hisshy：A pythagora switch is a device where one single point of failure ruins the whole thing. That’s what our image search feature was (laughs). In a CtoC service, the users are the ones who provide the search results and ensure that features are being used to their full potential. We were pursuing ways to create a reliable and positive experience for users in spite of these difficulties.

Why we named it “image search.”

@hisshy：To elaborate more, “object recognition” was a major part of this feature aimed to provide customers with a positive experience. After taking a photo, users can select which item in the photo they want to search for. This allows users to specify one single item, even if there are multiple items in the photo.

Using the object recognition feature, users can select exactly which item they want to search for.

ーBeing able to select the item by yourself sounds great. Was this being considered from the very start?

@hisshy：Originally, this feature didn’t allow users to select specific objects. Having the user select an item before searching would have been, to put it frankly, too difficult and costly to implement in the UI. However, our product managers gave us some feedback that got us thinking. “It’d be unfortunate and inconvenient if the user went through the trouble of taking a photo, only to have the app search for the wrong item.” “Would we really be providing an enjoyable buying experience by making them take photos repeatedly?” After considering these issues, we ended up discussing it with the developers and including it in the UI, shortly before the feature’s release.

ーSo this was a last-minute decision?!

@hisshy：Yep. We really did everything we could to make the experience as enjoyable as possible for users. After changing the UI, the feature received a considerable amount of praise from the product managers. We were really happy about that.

@metalunk：We also were very particular when choosing methods for item detection. I mentioned before that Mercari is a CtoC service dependent on the users, so if 1 million people list items, there will be at least 1 million photos available. The thing is, there are different ways to take photos of the exact same item. For clothing, some users might take a picture of it on a hanger, while others might lay it on the floor. There are a lot of different possibilities.

ーThere are users who take pictures of their clothes while wearing them, right?

@metalunk：That’s right. It’s really hard to make the feature recognize exactly which item to single out. When a person takes a picture of themselves wearing the article of clothing, the feature might pick up the person instead. Fortunately, Takuma Yamaguchi and several other members of the AI Engineering Team made it possible to detect just the targeted article of clothing. ((Our company has applied for a patent for the technology used by certain parts of the feature (Patent Application 2018-220243)))

ーBy the way, why did you settle on “image search” as the feature’s name?

@hisshy：We decided to do this after conducting user interviews. During the interviews, we presented a hypothetical scenario before having them try out the feature. With the image search feature, we used the scenario of “my friend told me about Mercari’s image search feature so I want to try it out.” For some reason, we weren’t sure what to call it. On Instagram, everyone says things like “this is a great photo,” right?

ーThat’s true. People do say that.

@hisshy：There may be a specific term that is considered mainstream around the world, but we felt that “image search” would be more approachable for the users. Someone the engineers in the office asked us why we chose that name though (laughs). But after releasing it, more and more people agreed that the term image search was appropriate.

The feature has been released, but this is just the beginning!

ーBy aiming to constantly keep the search results updated, the initial release means that you’ve only just gotten started. What are the future plans for this feature?

@hisshy：We’re glad to say that we’ve been noticed by a lot of people following the feature’s release. However, there actually aren’t that many people using the feature on the app itself. Some users might not fully understand the purpose of image search, so we want to spread awareness while helping those users get accustomed to it in a natural way.

@metalunk：As part of the AI Engineering Team, I want to increase the search accuracy as much as possible. For updating the search results, I think it’d be great if we could create a system that takes even the most recently listed items and makes them available as search results.

@hisshy：As a product manager, I have 1 request for @metalunk and the other AI Engineering members.

@metalunk：Sure! What is it?

@hisshy：Completing an image search still takes a bit of time compared to a normal text-based search. Of course, we can’t do much about the fact that reading image data takes time, but I think that there’s still room to test out new methods for speeding up the process. The better the speed, the more users there will be who want to use the feature. I’d love to see you and your team take image search to the next level!

@metalunk：Leave it to us! We’ll be doing everything we can to make the feature better and better!

プロフィール

Ryusuke Chiba

Joined GREE as a new grad, working on backend development for GREE News and the game Shoumetsu Toshi. Went to graduate school afterward to research mathematical optimization. Joined Mercari in 2017, working for the Search Team and later moving to his current position in SysML as a developer for the image search feature. With an insatiable appetite for learning, he actively participates in mathematics departments, doctorate programs for working adults, and company-run club activities.

Yasuo Hishii

Started his career at Fujitsu as an infrastructure engineer. Later worked at an IT startup, gaining experience as an app engineer and product manager. Joined Mercari in July 2018, and currently works on feature development and analysis as a product manager.