In addition to implementing generative AI in our products, at Mercari we use these technologies internally to improve our work and the productivity of our members. However, we are also acutely aware that inputting company information that has not been released to the public into any of the generative AI services now available online comes with a risk of information leaks.

To mitigate this risk while allowing our employees to use generative artificial intelligence and large language model (AI/LLM) technologies such as ChatGPT and Gemini in their work, in April 2023 we released “Ellie,” a tool developed to operate internally at Mercari. In charge of developing this tool was our Eliza team, a small collective of AI professionals tasked with the mission of implementing LLM technologies to dramatically maximize business impact and improve the productivity of the entire company. For this edition of Mercan, we spoke with Eliza team members Sho Akiyama (@akiyamasho), Yuki Yada (@arr0w), and Tzu Huang (@zoo) about the developments that led to the internal creation of a generative AI tool, their sense of the current problems with the technology, and their future outlook.

We were aiming to create a playground-like environment where employees could easily try out AI/LLM technologies. 

—To kick off our conversation, could you talk about what led to the development of Ellie?

@arr0w：The principal factor behind our idea to develop Ellie was that we thought it would be ideal to create a playground-like environment where employees could easily try out AI/LLM technologies. Going forward, we’re also thinking about implementing features that will respond to questions based on knowledge and context regarding Mercari, so we have decided to create this inhouse.

@akiyamasho：Of course, we wanted to do more than just create a playground. Our intention was to improve our actual development environment and optimize our work at Mercari.

@zoo：We also wanted to create an opportunity as a company for our members to learn about LLMs. It gave us a chance to uncover the kinds of issues we could resolve using this technology. Moving forward, we’d like to be able to pinpoint which of the different types of LLMs are most appropriate to use depending on the issue.

—So what features does Ellie actually offer its users?

@zoo：Currently, Ellie uses APIs for ChatGPT (OpenAI), Gemini (Google), and Claude (Anthropic) and allows members to use the general features of an AI chatbot in a safe environment. Specifically, you can use Ellie to do things like have brainstorming sessions (in a chat format), upload various types of files, search the web, and translate text.
Employees can of course generate images as well as share their prompts with their colleagues and use the prompt templates in our prompt library. And since Ellie supports the latest models, employees can choose to use the model they prefer.

I think an additional selling point is the “super private” mode we’ve set up that allows employees to use this tool without their conversations being logged, which enables them to work with more sensitive information.

—So what’s your impression of how things have been going since you released Ellie? What issues do you feel could come up?

@arr0w：That’s easy; the biggest issue remains how to spread these technologies within the company. Whenever we implement a new feature, we announce it on the project Slack channel we created for the Ellie project, but spreading the word has proven tough. In recent months Gomi Hayakawa (@gomichan) has led LLM education sessions at the company, and our Eliza team has also been posting tips on how to use the tools.

We also analyze usage trends, and every week usage data is posted on Slack automatically, but another issue we face is how to discern what new features we should create based on this usage data. In addition, when we analyze usage trends, we use LLMs to protect people’s privacy; the entire process is automated.

@zoo：Personally, I don’t think we’re dealing with any big issues involving the actual development of Ellie; however, I do want us to avoid reinventing the proverbial wheel. For this reason, we’ve had some difficulties discerning where within the company we need to use AI, what we should prioritize, and how to rigorously manage our resources while keeping an eye on the evolving situation of generative AI technologies outside the company.

—I see. So where Ellie is concerned, what sorts of things have you prioritized for development?

@zoo：The feature that was most in demand was one that would allow members to create their own unique chatbots based on the documents and information of their team. We created a proof of concept (PoC) for this feature and found that there is a certain amount of demand for it, and so we plan to implement it in the future.

Usage rates are expanding steadily. Now the question is, what should we focus on next?

—How have usage rates changed since you released Ellie?

@zoo：We started development in March 2023 and released Ellie about two or three weeks later. Up until about August 2023, usage rates climbed gently. Then in September 2023, when we released various features like translation and private mode, usage rates suddenly spiked to nearly 40%. Currently, usage rates have climbed to 75%, and nearly 1,600 employees use Ellie every month. These are precisely the numbers we set in our original OKR. By the end of 2024, I think I’d like to get the usage rate up to 80%.

—How do you collect feedback from employees and use it to make feature improvements?

@zoo：Basically, in addition to collecting information using Google Forms, our team receives feedback directly on Slack.

@arr0w：To maintain people’s privacy, we perform analysis using LLMs and categorize each message and chat according to its purpose. What we found in our results was that because Mercari is a tech company and a global enterprise, a lot of our people use Ellie for translation and coding. Personally, I think it’s important to focus on analysis, and the GPTs[1] feature is also important.

* [1] GPTs: Custom versions of ChatGPT that users can tailor for specific tasks or topics by combining instructions, knowledge, and capabilities.

@zoo：There is of course a lot of documentation for each team within the company. In the future, I’d personally like for there to be a feature that allows you to not just search for information, but also acquire a variety of information using a chatbot. And it’s not just engineers who want to use Ellie to compile their documents either.

In the feedback we’ve received from our members, we’ve also had requests related to UI improvements, like requests asking us to place buttons in specific locations, so I’d like us to implement those improvements. We were really rushing when we developed and released Ellie, so I can’t say that the UI and UX are optimized. So far we’ve interviewed at least 100 members, and a lot of people don’t know that they can switch between the various different models of a variety of LLM services, so I want to make an announcement about this so people get the message.

@akiyamasho：We’ve managed to get a lot of our members to use Ellie, but it’s still hard to gauge the effect it’s having. For example, it’s hard to measure how much time it saves people everyday.

@arr0w：It’s sort of hard to measure the quantitative impact it has. It’s an entirely different conversation from pointing to a number and saying, this is how much Ellie has contributed to the company’s profits.

@zoo：To solve this problem, we tried a number of scientific methods to show how much people had increased their productivity using Ellie, but it did not go well. This is when we thought to investigate OpenAI and Microsoft use cases. They had also done user interviews, and their employees had answered subjectively about how much time they had saved (by using generative AI). We would also like to start making similar time measurements.

I want Ellie to become a platform that brings together various AI tools

—Finally, I’d like to ask you about what the Eliza team would like to accomplish that includes Ellie. Do you have any hints or spoilers about what’s coming up?

@zoo：Currently, we have an issue that there are a lot of people at the company who don’t realize that the tool’s name is Ellie. People call it different names, so one of the things we’d like to do is popularize Ellie’s name. Next, we’d of course like to make Ellie a hub that brings together various AI tools.

There are two types of use cases: Pre-Generated Content (PGC) and User Generated Content (UGC). For PGC, the Eliza team would like to confirm use cases beforehand and deliver the use cases to our users.

However, when it comes to UGC, this describes when individual members create chatbots to use AI. Because development takes place in an environment that employees can manage themselves, by linking with Google Workspace they can create a folder in Google Drive that is just for them or just for their team.

For example, one team uses Ellie as a sounding board for HR evaluations when looking back on their own work in order to write fairer and more effective self-evaluations. I would like to make this available to a variety of other members throughout the company going forward. You don’t need to know coding; it’s just a matter of uploading your document and then copying and pasting a prompt into Ellie, so even non-engineers can use these features.

@akiyamasho：As the Eliza team expands, I think there are three important elements to consider. One is achieving productivity similar to Ellie’s. Another would be something like the listing tool that we released for Mercari users in September called AI Listing Support. The last thing would be the work efficiency of the ML teams across the entire company. To that end, just as @zoo said, we want to establish Ellie as a platform that brings together various AI tools.

@arr0w：Personally, I think activities geared toward improving LLM literacy at the entire company are important, so I’d like to keep putting out that type of message. Additionally, I’d like to submit a paper to the EMNLP (Empirical Methods in Natural Language Processing) conference, which is said to be the most rigorous conference in the entire field of language processing. I’d also like to proactively send out information to academia.