On September 10, 2024, Mercari released a new feature called AI Listing Support that automatically fills out item information after a user takes a photo of the item they want to list and selects an item category.

This feature was born out of a desire to make the listing experience even easier, eliminating steps such as writing an item description and setting a price to address pain points in the listing process expressed by many people in user surveys.

All the lister has to do is take a picture of an item and choose the appropriate category. The item description, price, and other necessary information is automatically generated with AI, enabling the process to be completed in as few as three taps.

In this interview, we spoke to project manager Ayaka Okubo (@ayeong) and machine learning engineer Andre Rusli (@andre) about the process leading up to the release of AI Listing Support, as well as planned updates for the future.

Using AI technology to overhaul listing on Mercari and solve the problem of the overly complicated listing form

──What was the thought process behind developing and releasing AI Listing Support?

@ayeong：In recent years, different development teams have added new features to the listing form to support users in listing items—Easy Listing, No Price Listings, Barcode Listing, and more.

But even though these features were intended to make listing easier, they actually made the listing form more complicated—to users without much experience listing items, like people using Mercari for the first time, the number of fields on the listing form made the process look more intimidating. As a seller on Mercari myself, I could tell that this was something that needed to be improved.

AI Listing Support was developed as a new automatic input feature to address this problem.

@andre：The engineering team was also aware that we had been receiving feedback from users asking us to simplify the listing process for a long time.

In 2023, with the rapid developments in AI/LLM technology, we began testing ways to use it to make listing easier for actual implementation. But we didn’t start developing AI Listing Support right away, because we needed to get an idea of what would and wouldn’t work well with AI first.
The first project we worked on with @ayeong’s team in charge of listing was a “seller hint” feature. For items that don’t sell within two or three days, this feature suggests ways the seller can edit the listing to make it more appealing to potential buyers.

The feature was one of the Machine Learning Team’s experiments, developed to support users while also gathering data about what items sell and what kind of information is missing from items that don’t. Once we had accumulated enough data, @ayeong came to us with a proposal for a prototype that would eventually become AI Listing Support, and we began working together to develop it.

──What kind of feature were you picturing for the prototype?

@ayeong：At first, we were just envisioning a feature that would automatically generate item descriptions when listing. But the more we thought about it, the more we realized that this feature could have an even bigger impact, and so we decided to try creating the feature at the scale it is now. When we tried it out using Ellie (an internal tool enabling employees to use AI engines such as ChatGPT and Gemini), the descriptions it generated were already accurate enough to be practical.

The demand for “AI” features…or not?

──In development, were there any parts that you found particularly difficult or that you particularly worked to perfect? I’d like to hear from both the user experience perspective and the engineering perspective.

@ayeong：From the user experience perspective, the key to this project was aiming to create an experience so obviously easy that users would feel it the instant they used the feature. We were careful to design this feature so users would seamlessly use AI as part of the listing process, rather than just adding AI on top of the existing listing features.

@andre：From the engineering perspective, there were two aspects that were particularly difficult. The first was that, as much as we want to make listing as simple as possible, it isn’t realistic to automate everything. In deciding whether to leave the step of selecting a category to the user, we analyzed data from our various experiments and the accuracy of the AI suggestions. In the end, we determined that having the user choose the category struck the right balance between simplicity and accuracy.

The second was handling hallucinations—cases where generative AI outputs information that is misleading or incorrect and presents that information as fact. For example, AI sees a picture of a silver laptop and automatically tags it as an Apple product. While that’s true in some cases, it’s not true in all. But if we try to avoid this by placing too many restrictions on AI output, then we also lose opportunities where it would have guessed correctly. So in the end, we designed some systems to detect and deal with hallucinations, and fine-tuned them while checking the output manually. There were some categories that we couldn’t cover with just one or two engineers, so we got some help from other experienced members within the company.

──Why did you decide to leave the category selection to the user?

@ayeong：We ran into cases where, say, the picture was of a toy car, but AI generated listing information as if it were a real used car. The category of the item is important information to ensure that the automatic suggestions are accurate. Mercari also has a category suggestion feature, which enables sellers to find the most appropriate category for their item with just one tap, so we thought that it wouldn’t be too difficult for the user to handle this step.

──Did you decide on the name AI Listing Support early on in the development process?

@ayeong：It actually took us a while to decide on the name. We went back and forth on whether to make the name catchy and easy to understand or make the name really unique, and had a lot of ideas for both. In the end, we came to the conclusion that we should go with a name that succinctly describes what the feature does. But to be perfectly honest, we’ve already been running some A/B tests for renaming it…

After A/B testing, we changed the wording on the app to “Automatically fill in the item name and description,” which had the highest usage rate in the tests. We came up with this wording based on the hypothesis that Mercari users may be more willing to try the feature if it didn’t use the term “AI.”

──That wording does sound like it would be more appealing to people who are reluctant to use AI. It also makes it more clear what exactly the feature does. So, is the ultimate goal of this project to increase the number of listers and listings?

@andre：That’s a good question. The ML Team sees the ultimate goal as supporting users in all steps of the selling process, from the time they list the item to the time it sells. But when using AI/LLM technology, there are so many things to consider—accuracy, the user experience, impact on the business, and more. So we can’t be the sole judge of whether what we’re doing is sufficient or not.

That’s why it’s important for us to work with teams who have plenty of domain knowledge, like the Discovery Team, the Search Team, and @ayeong’s Listing Team.

Perfecting technology to enhance the user experience

──What can we expect to see from AI Listing Support in the future?

@ayeong：We plan to steadily update the UX to make the experience more interactive. For example, we want the UX to naturally guide the user to take pictures that help AI generate better information.

@andre：There are still many parts of the listing process that we can improve. We finally broke down the barrier to listing using AI technology, but we can do better to lay the path that extends ahead. I’d also like to expand AI features to create a better experience for users searching and buying items, and to help listed items find their way into the hands of users that need them.

──Are there any improvements you’d like to make to the automatically generated text?

@ayeong：We want to get rid of “AI-sounding text.” We’ve already received feedback from users about the generated text sounding mechanical and not how a human would write. Going forward, we hope to implement personalization based on the seller’s writing style to encourage more people to use it.

@andre：I see the current state as about halfway to where we want it to be. There’s still a lot to work on. The field of AI is constantly changing, and the technology may be able to do even better things six months from now. Our goal is not only to make the product easier to use, but to carry out small improvements over time to enhance the user experience.