Why did candidates who said yes more often make worse matches?
Acceptance alone selected weaker two-sided matches. I reconstructed historical decisions, compared selection policies, and implemented composite selection with simulation limits made explicit.
The initial problem
In a subscription matching product, a selection group reserved four slots for candidates with high historical acceptance and four for a composite score—a score combining multiple selection signals. The assumption was reasonable enough to test: people who said yes more often should produce more successful matches.
But a candidate's willingness to accept is only one side of a successful match. The other party must want the match too. I needed to determine whether the selection rule was improving that shared outcome.
How I approached it
I reconstructed historical decisions using only information available before each event. A sensitivity check added a lag for delayed responses, reducing the risk that future outcomes would leak into the reconstructed score.
Candidates with the highest acceptance rates had weaker interest from the other party. Selecting for willingness alone favored an edge case that the operating objective did not reward. The historical analysis found approximately 1.4× lift for each individual gate, or filtering condition, and 2.72× for their intersection—candidates satisfying both conditions. Those were associations in the analyzed data, rather than a causal experiment.
I then compared selection policies, not just scores. Simulations with different candidate-pool sizes let me examine both the success rate and the absolute number of successful matches. A harder threshold could raise the rate while leaving slots empty; in a thin pool, that reduced total successes.
How I solved it
I compared the existing four acceptance-only slots plus four composite slots with eight composite slots. In the simulations, selecting eight by the composite score increased successful outcomes by 17–39% relative to the prior policy, without reducing the number selected in those comparisons.
The composite selection policy was implemented. The minimum eligibility threshold remained a separate constraint and can still leave capacity unfilled. That distinction matters: changing how eligible candidates are ranked does not guarantee that every pool contains enough eligible candidates.
What the evidence supports
The 17–39% improvement is a historical backtest and selection simulation, not measured post-change production lift. Candidate pools were simulated, and the source report did not yet have matured post-change outcomes.
The next step is a cohort comparison that includes pool size, selection capacity, and absolute successful matches. The work changed the selection rule while keeping the evidence in its proper category: historical association, simulated comparison, and implemented policy.