Recently, Mercari’s Business Intelligence Product Team launched an internal analytics tool called “Socrates” designed to help members make faster, high-quality decisions.

Socrates is an analytics tool that uses generative AI to give members access to the data they need to make decisions without having to ask an analyst. With the release of this tool, data analyst resources no longer become a bottleneck, and anyone can readily find the data they need.

In this Mercan article, we spoke to members of the Business Intelligence (BI) Product Team responsible for the development of Socrates—manager Kentaro Kobayashi (@kobaken), PdM director Yasunori Hishii (@hisshy), and product manager Shoma Ishimoto (@shouma)—about how Socrates is being put to use and the changes it has brought about.

Lowering requirement definition costs and eliminating the “guilt” of making requests

—To start, could you tell us how you are currently using Socrates?

@shouma: Before Socrates was released, decision-making involved exploring data, such as the number of users of a certain feature or the long-term trends of certain metrics, by investigating log specifications, writing queries, and compiling results.

Now that we have Socrates, it is much easier to gather the data surrounding each decision we need to make, which has helped us out immensely. We just have to write a prompt of about one line, and Socrates outputs the data we are looking for. This has improved our work efficiency and enhanced the speed and quality of our decision-making.

Also, for exploratory analysis where there might be more than a single answer, Socrates makes things dramatically more efficient. For example, just say we wanted to compare how the search function of the app is being used by two different user groups: frequent Mercari users, and users who signed up recently. All we have to do is input a prompt of about three lines, and Socrates will analyze data, create a detailed report, and provide a breakdown of the report using the same approaches we traditionally use when working with analysts. Socrates then goes one step further and suggests the next phase of analysis based on the results. Wonderful, isn’t it?

—What kind of prompts do you write specifically?

@shouma: In the scenario I just mentioned, the prompt would be: “Analyze trends in search function usage based on users’ purchasing behavior.” Of course, Socrates may misunderstand what I’m looking for, so I sometimes have to alter the prompt and rerun the model based on the output.

When working with analysts, differences in understanding can cause unnecessary delays, so it’s crucial to determine analysis requirements upfront. With Socrates, I don’t feel any guilt about not communicating well or being asked to clarify my intentions because I phrased something in a confusing way. If the model doesn’t quite understand what I mean, I can quickly make course corrections. This significantly lowers the psychological barrier that comes with data analysis.

@kobaken: Even if we had all the analysts in the world, I think people would still be hesitant to send them requests—that feeling of not wanting to bother people tends to persist. With Socrates, we can define the requirements through talking with the model, so it doesn’t matter if our initial requirements are vague. And if people are using Socrates, analyst resources can be redirected to higher-value work, making it easier to generate additional value.

@hisshy: Socrates is also a great tool for addressing confirmation bias. Those with experience in data analysis may have noticed that in the course of working on a project, some people have unconscious bias—that is, they only gather information that supports their hypothesis and choose not to consider variables that might lead to negative results. Since Socrates doesn’t have any preconceived notions about what the outcome should be, it gives unbiased responses about anything that needs clarifying. So, Socrates also represents a major advance in eliminating analytical bias.

The freedom of finding facts without talk of priority or purpose

@kobaken: @shouma, I think your background as an analyst enables you to use Socrates in such a flexible way. Do you think members who don’t have the same experience are also getting the same benefits from Socrates?

@shouma: That’s true; having analytical experience probably makes Socrates easier to use. We have a UX designer on the team who has no analytical experience, and they struggled to use Socrates. They told me that they wanted to come up with a UX design based on an analysis of the search function, but they couldn’t get Socrates to output the right information.

This made me realize that people with analytical experience are more accustomed to writing prompts, whereas those in other roles may still need support. Increasing our team’s overall prompt writing skills will be something for us to work on going forward.

@hisshy: I think some practice is required to use Socrates effectively. But, I feel that the basic concept of Socrates provides significant value even to those without an analysis background. Just say that I had a spur-of-the-moment idea to investigate something from a certain angle and wanted to begin exploratory analysis. Teaming up with an analyst would bring questions of priority and value into play, so it would be pretty much impossible to test out every idea I had. But with Socrates, I can take the time to have a long conversation with it and test all sorts of ideas.

So, I think the true value of Socrates lies in not just speeding up our work, but also making things possible that were previously impossible. To me, this is a significant change on par with a paradigm shift.

Also, it’s amazing that Socrates allows us to search information across teams. Before, I would find the information I wanted by cherry-picking features from different in-house tools; with Socrates, it’s all in one place.

For example, we currently track usage trends separately—one set for individual Mercari users and another for businesses selling on Mercari Shops. With Socrates, we can examine all of this data across the marketplace as a whole, which could reveal issues and insights we hadn’t noticed before.

I suspect that other Mercari services, such as XB (crossborder transactions) and NFTs, influence each other, and using Socrates to provide numerical insights into these relationships could positively impact the overall strategy of these services.

@kobaken: In terms of strategy, the information assets held by each team are extremely important. In the past, running a cross-functional analysis meant tracking down the data owners in every team. Going forward, I want Socrates to be able to retrieve all that information for me with a simple prompt.

For example, if we let Socrates read the specification sheets, logs, and other documents maintained by each team, that material will become part of its analytical context, and it should be able to deliver even deeper insights. I think we need to build that kind of data environment in collaboration with every team, rather than keeping it siloed within the teams that regularly handle data. That will be an initiative we will pursue in parallel with developing Socrates.

Aiming for a reality where anyone can analyze information

—It sounds like Socrates will continue to evolve as Mercari Group continues to grow. Lastly, could you tell us what your ideal version of Socrates looks like?

@kobaken: I hope that people see Socrates as more than just a convenient tool. Ideally, it would become known within the company as a sort of “coworker” that helps us humans expand our abilities. The ideal relationship between humans and AI is one where we complement each other’s strengths and weaknesses so we can make high-quality decisions more quickly.

For example, we could use Socrates to do a task that would traditionally take a human dozens of hours, such as creating a draft of an analysis report. With Socrates, the report would be about 80% complete, and then us humans could focus on polishing the remaining 20%. When collaboration between humans and AI becomes the norm like this, the overall productivity of the organization should dramatically improve.

@shouma: We want anyone—regardless of their access to analytics resources—to be able to gather user insights for any purpose, so they can make data-driven decisions and move projects forward. In my view, Socrates is on the verge of being able to replace nearly all of the work done by PMs and analysts. If that happens, we can then dedicate more time to exploratory analysis, which has been deemed a low priority, opening the door to new hypotheses and proposals. I’d like to see Socrates keep evolving as a tool that helps people realize their full potential.

@kobaken: Now that we leverage AI/LLMs more for data analysis, we are placing more importance on data management and data modeling. There are ample opportunities in Mercari Group to use data, and teams that haven’t yet been able to maximize that potential are asking to use Socrates. Requests like that serve as motivation for us to move the development of the data infrastructure forward. We want to create a positive circulation of value where we use the data infrastructure to make Socrates even more powerful and accelerate data usage across the organization, so that every member will be able to use Socrates to help them with their needs.

@hisshy: I mentioned earlier the need to break away from data silos. We aim to make Socrates a catalyst that will improve access to all forms of data, remove the domain barriers that stifle cross-functional ideas, and ultimately accelerate Mercari Group’s ability to maximize user value.

Photographer: Tomohiro Takeshita / Author: Yuta Ishikawa / English Translation: Mercari Global Operations Team