At the on-demand work service Mercari Hallo, our concept of work is simple: “quick and easy for everyone.” With this in mind, our service connects partners (companies) urgently looking for workers with crew (workers) urgently looking for work. Just over a year since our March 2024 launch, over 10 million people have registered as crew for our on-demand work service, making us the number-one provider in the industry.

At Mercari Hallo, we have developed a feature to help make it easier for our partners to look for workers; we call it the “Easy Job Listing” feature. The feature allows our partners to create customized, appealing job listings with just a few simple clicks and some light typing. Behind the curtain of the feature are the latest artificial intelligence (AI), machine learning (ML), and large language model (LLM) technologies.

For this article we spoke to job listing quality project leader Maiko Hiyama (@michael) and generative AI-related project members Tomoyuki Habu (@t-habu) and Yuki Yada (@arr0w) about the background and functionality of Mercari Hallo’s “Easy Job Listing” feature.

The triple win we achieved for issue resolution in our pursuit of convenience and a sense of excitement

——Could you start by talking a little about the advantages of the Easy Job Listing feature and the story behind its development?

@michael：Using Mercari Hallo’s Easy Job Listing feature allows our partners to create an appealing job listing in just seconds. All they have to do is answer a few multiple-choice questions and enter a few simple sentences into our platform. There’s no need for them to write anything at length or wrestle over what expressions to use to make their listing sound appealing. The feature makes it easy for partners to create original job listings that easily attract applicants.

@t-habu: It can take anywhere from 30 minutes to an hour for a person to write a job listing from scratch. Users who are new to the process can take even longer. Using the Easy Job Listing feature, they can generate a rough draft in seconds, shortening the time it takes to create a high-quality job listing to about a minute. If you think of this in terms of personnel expenses, it translates to a cost savings of about 1/100 of the usual expense.

——What specific points make our job listing feature different from those offered by other companies?

@t-habu：The job listing services offered by other companies are largely based around templates. The pitfall with using templates is that the listings end up looking the same due to using the same elements in listings for the same job types. Crew looking for work are unable to distinguish job listings based on the job description and end up making their decisions based only on factors like the hourly wage and how far the workplace is.

As the service operators, we of course want the crew who read job listings to also feel excited about using the service we provide. Providing a service that offers a rich variety of listings makes for a more interesting experience, and users will spend more time on our service as well. This is why we set out to use the power of AI and LLM technologies to create job listings that would draw out the individuality of our partners.

@arr0w：One of the major traits of the Easy Job Listing feature is that listings are personalized for your company. Partners can register descriptions of the duties crew will perform and other qualitative information, such as the atmosphere of the workplace, to create appealing job listings that reflect these traits. I think it’s a big benefit that users can experience creating attractive job listings with so little effort.

@michael：The result is that we’ve made three groups happy: Our partners who create listings, the crew who look for work, and ourselves, who want to increase the overall quality of job listings. It’s like we’ve achieved a triple win!

The best solution for tackling the issue just happened to be AI/LLMs

——So, what was the sequence of events that led to you talking about using AI/LLMs to create the Easy Job Listing feature?

@michael：I guess you could say that it all started with the collaboration project between the AI/LLM Team (known internally as the Eliza Team) and Mercari Hallo, but now we’re working on it independently.

From what we could tell, there were three issues with the quality of Mercari Hallo job listings created the old-fashioned way.

First, people end up using prohibited expressions* that we then have to edit to create text of a quality suitable for posting, which takes time. Second, there are only so many people capable of writing high-quality job listings good enough to have an impact on application rates, so this meant that we ended up depending on specific individuals. Third and finally, for job types with similar job descriptions, using a template-based mechanism to write job listings ends up producing generic job listings that look and sound the same.

*This refers to regulations for creating job listings that prohibit the use of certain terms, including expressions forbidden by laws and ordinances, in order to ensure crew are safe and secure at work.
https://help.hallo.mercari.com/–656ba2b874f4b4002423379a (available only in Japanese)

LLMs are the one technology that is able to resolve all three of these issues. Sometime in the spring of 2024, we decided to use LLMs with the goal of eliminating any negative aspects of our service.

@t-habu：Excitement over AI has increased recently, so the technology tends to be front and center, but within our project we were consistent in our insistence on a user experience that did not feel like a direct interaction with AI. Some of the people who use our Easy Job Listing feature are not adept at using AI and other technologies. All they know is that AI and LLM technologies are working behind the UX while they make a few simple clicks to create their job listings.

@arr0w：LLMs are a clever technology that allow users to generate sentences tailored to their needs. Since we had the technology available, we moved to use it to create added value, and we had something similar to a task force that kicked off from the bottom up.

——Could you explain in more detail how the Easy Job Listing feature increased the quality of Mercari Hallo job listings?

@t-habu：I think we have to start by defining what a “good” job listing is in the first place. We started by looking at job listings and analyzing the shared traits of the ones that had a high match rate. We broke the segments down according to a variety of elements like sentence structure and items that are easy to evaluate.

The factors that impacted conversion rates and match rates were more than just the content of the job listings. Of course, there were elements that likely had more of an impact like hourly wages and conditions. We analyzed past job listings to see what would happen when we eliminated quantitative information like pay.

@michael：We narrowed our focus while coming up with a variety of hypotheses. We conducted comparison and verification tests between job postings created manually by our partners and others created by AI based on writing styles we evaluated as good, and then we leveraged the better patterns.

In the beginning, we didn’t know which would be the best metrics to examine. We observed the impact from a variety of aspects and incorporated a job listing definition over time and through several rounds of QA. This is where the human element worked hard.

@arr0w：In the end, we defined the job listing evaluation metrics in three stages. Among these, in order to generate listings of a quality that would satisfy the top criteria, we fine-tuned the logic for the Easy Job Listing feature. Finally, we implemented LLMs as a judge that was able to automatically evaluate job listings using LLM technology and performed tests to increase its accuracy.

Continuing to expand the user experience was of the utmost importance

——What features and prospects of Easy Job Listing would you like to spend time on going forward?

@t-habu：I’d have to say, the images generated on the top page of listings. The images arranged on the timeline are another important element that greatly impacts application rates. It’s already possible to generate images using LLM technology, but I would like us to challenge ourselves to further improve quality and the user experience.

@arr0w：In addition, I’d also like us to work on expanding the input fields. The only input fields that a user can select are the default items provided for things like wages. The information that you can enter as open comments in fields like “Work description” could also be suggested from selectable categories. I think we can make this more convenient to use.

@arr0w：As for me, I’d like to create a recommendation feature that further expands the options that crew have.

Imagine, for example, that there’s a person who is thinking about working at a café, and they start searching for such a job. Although they may not realize it, they could actually be a good fit for working at a bar. I think it would be great if we could create matches by getting information to the candidate crew who is a potential good fit for that bar job, but who isn’t actively looking for it.

@t-habu：It would be great if we could get AI to make suggestions as a sort of consultant. The feature would also work with the partner side by providing advice about how to write postings that attract crew who are not actively looking for them.

@michael：That’s a good point. This brings to mind the experience of going to a music festival where you end up liking the opening and closing acts of the band you’re there to see. AI can present you with suggestions for options unlike anything before. I’d like to see us provide that experience.