A Concrete Moment: The Slack Thread That Sparked Change
During a recent Slack thread, one of our recruiters shared frustration over mismatched candidate profiles. A candidate with strong qualifications was being overlooked for roles that seemed to fit perfectly. This sparked a discussion around our AI candidate matching algorithm and its shortcomings. We knew we had to dig deeper into the mechanics behind our matching percentage.
Understanding the Problem
The importance of effective candidate matching cannot be overstated. For our recruiters, the stakes are high; their success hinges on presenting the right candidates to companies. Misalignment not only frustrates recruiters but also jeopardizes candidate opportunities. We felt this pressure acutely as we aimed to streamline the hiring process for both candidates and companies using our platform.
The Core Issue: Mismatched Skills
A specific scenario highlighted our problem: a senior software engineer was passed over for a role due to our algorithm misinterpreting their skills. Our algorithm struggled with skill synonyms; terms like “software development” and “programming” didn’t always trigger the same match score, leading to confusion. This misalignment meant that qualified candidates were not being surfaced in relevant job listings, and our matching system was failing to deliver value.
Initial Attempts: False Starts and Setbacks
In our initial attempts to remedy the situation, we explored simply adding more deterministic filters based on hard-coded skill sets. This seemed straightforward but ultimately didn’t address the underlying issue of semantic understanding. One of our backend engineers quickly identified that this approach limited flexibility and didn’t account for the nuances of human language. We realized we needed a more sophisticated solution.
A New Technical Approach: LLM Re-ranking
We pivoted to implementing a large language model (LLM) for re-ranking candidates after an initial match retrieval phase. The process involves two main steps:
- Retrieval: Candidates are initially matched based on deterministic filters—skills, experience, and education.
- Re-ranking: The LLM then assesses the matches, considering synonyms, context, and relevance to provide a final score.
# Example of re-ranking with LLM
re_ranked_candidates = llm_model.rank(candidates, job_description)This method dramatically improved our ability to surface candidates whose skills might have otherwise been overlooked due to terminology differences.
Observable Changes in Our Product
After deploying the new algorithm, we observed significant changes. Candidates began receiving more relevant job recommendations, enhancing their experience on our platform. Recruiters also reported increased satisfaction with the quality of candidate matches, leading to quicker placements and reduced time-to-hire metrics. These improvements can be seen on our /jobs and /for-companies pages, where recruiters now have access to a more refined candidate pool.
Key Takeaways from Our Experience
- Language matters: Synonyms and variations in skill descriptions can affect matching outcomes significantly.
- Flexibility is crucial: Relying solely on deterministic filters can limit the algorithm's adaptability to real-world language use.
- User feedback is invaluable: Direct input from recruiters and candidates helped us identify issues that we might have missed initially.
Implications for Candidates
From a candidate's perspective, this improvement means a better chance of being matched to roles where they truly fit. Candidates can expect to see opportunities that align more closely with their skills and experiences, leading to a more satisfying job search process.
Implications for Recruiters
Recruiters can now rely on a more robust matching algorithm that takes into account the nuances of language and candidate profiles. This means enhanced efficiency in sourcing candidates, ultimately benefiting their hiring goals and the companies they serve.
Looking Ahead: Open Questions and Future Directions
While we've made significant strides, we recognize there’s still work to be done. We’re monitoring for persistent issues like seniority inflation and keyword stuffing, which can still skew match percentages. If we had to undo one aspect, it would likely be the overly rigid initial filters that limited our flexibility. We’re committed to refining our approach and exploring how we can enhance our algorithm even further.