---
url: https://pmfolio.xyz/demo/search-relevance
title: Search satisfaction NPS from 52 to 79 — and CTR on top result up 34% — pmFolio demo
description: ShopCo's B2B marketplace had a search engine that was essentially a keyword matcher built in 2017. As the catalogue grew to 4.2M SKUs, keyword search broke down
type: pm-case-study-demo
---

# Search satisfaction NPS from 52 to 79 — and CTR on top result up 34%

**Company**: ShopCo
**Role**: Senior PM
**Duration**: 8 months
**Team**: 8-person squad (3 ML, 2 BE, 1 FE, 1 data, 1 PM)
**Skills**: ML, Search, B2B, Platform, NLP

## Key Outcomes

- Search satisfaction NPS improved from 52 to 79
- Click-through rate on top search result up 34%
- Zero-result searches reduced from 18% to 6% of queries
- Search-initiated GMV up 22% quarter-on-quarter

## 01 — The Problem

ShopCo's B2B marketplace had a search engine that was essentially a keyword matcher built in 2017. As the catalogue grew to 4.2M SKUs, keyword search broke down: synonym blindness, no understanding of intent, and no personalisation for repeat buyers. 18% of searches returned zero results. Buyers were switching to Google to find products on ShopCo — then coming back to complete the purchase.

> Search was the #1 product discovery channel, accounting for 63% of all browse sessions. Buyers who used search converted at 2.8× the rate of browsing buyers.

## 02 — Research & Insights

**Methods**: Query log analysis (6 months, 40M queries), 22 user interviews with B2B buyers, 8 interviews with suppliers, 3 competitor benchmarks, monthly NPS survey segmented by search usage

Query log analysis revealed three dominant failure patterns: synonym failures (34% of zero-result queries had a successful equivalent in our logs); intent mismatch (queries containing 'cheap', 'bulk', 'sample' were not being interpreted as intent signals); and catalogue gaps (28% of zero-result queries were genuine gaps, not a relevance problem).

## 03 — Solution

We rebuilt relevance in three layers: a synonym and query expansion layer using a fine-tuned sentence transformer model; an intent classifier that detected modifier intent and adjusted ranking; and a personalisation layer that boosted previously-purchased and viewed SKUs for logged-in buyers.

### Key decisions & trade-offs

The biggest architectural decision was whether to use a general-purpose LLM or a smaller, fine-tuned model. The LLM approach was 40× more expensive per query at our volume and added 120ms to search latency. We chose the fine-tuned sentence transformer — smaller, faster, cheaper.

## 04 — Results

| Metric | Before | After | Delta | Timeframe |
|--------|--------|-------|-------|-----------|
| Search satisfaction NPS | 52 | 79 | +27 | 8 months |
| Top-result CTR | 31% | 42% | +34% | 8 months |
| Zero-result rate | 18% | 6% | −67% | 8 months |
| Search-initiated GMV | baseline | +22% | +22% | Q4 2024 |

## 05 — Challenges & Learnings

### Challenges

The intent classifier had a persistent precision problem on ambiguous queries. 'Office chairs' could mean budget intent or just browsing — the model was over-indexing on price signals from the training data. We had to build a feedback loop from explicit user signals to correct the bias.

### What I'd do differently

I should have scoped the catalogue gap work out of this project from the start. We spent 6 weeks building the gap signal pipeline, which delayed the relevance work. I also learned to always run a shadow evaluation before an A/B test for ML ranking changes.

---

*This is a demo case study built with pmFolio. Build your own: https://pmfolio.xyz*

*Back to demo portfolio: https://pmfolio.xyz/demo*