1. The Billion-Vector Problem

Say you’re running a search for an ecommerce marketplace with a billion product embeddings. The obvious instinct is to throw it all in RAM, both vectors,index, everything because RAM is fast and disk is scary. Then you price it out and the instinct changes fast.

An ecommerce marketplace with a billion product embeddings, each one at 384 dimensions, stored as float32.

The arithmetic is simple. Each vector costs:

384 dimensions × 4 bytes = 1,536 bytes per vector

Scale that to a billion vectors:

1,536 bytes × 1,000,000,000 ≈ 1.54 TB

These figures represent only the “naked” vectors. A production-ready environment faces significant “invisible” costs:

HNSW Graph Links: Connectivity data for navigation, which can add hundreds of gigabytes at billion-scale.

Payload Indexes & Write Amplification: Metadata for filtering and the overhead of segment compaction.

Replicas & Recovery: Data redundancy for high availability and disaster recovery.

Operating System Overhead: Page caches and temporary ingestion memory. Each of those adds its own RAM bill, and none of them are optional in a real deployment.

Each of those adds its own RAM bill, and none of them are optional in a real deployment.

So the obvious question: do you actually need 1.5+ TB (or, at the more commonly cited 768-dimension embedding size, closer to 3 TB) of RAM just to search a billion vectors?

The answer is no, but only if you design the storage and retrieval architecture on purpose. Billion-scale vector search isn’t really a RAM-sizing exercise. It’s a decision problem: what has to live in RAM, what can live on disk, and what representation should each stage of retrieval actually use. The rest of this post works through that decision, one lever at a time, then tests the results on a real benchmark.

2. Qdrant’s Storage Model

Given that requirements continuously evolve, why should we be forced into binary tradeoffs? A truly adaptable system should allow seamless customization to meet a product’s specific demands.This is where in my case, I came across Qdrant.

Qdrant is a natural fit for this problem because it treats memory placement as a per-component decision rather than an all-or-nothing switch; vectors, quantized vectors, the HNSW graph, and payload indexes can each be independently placed in RAM or on disk, which is exactly the kind of control a billion-vector system needs. Its quantization isn’t a separate compression step bolted on afterward; it’s integrated with the retrieval path itself, supporting native rescoring against original vectors with a tunable oversampling factor. On-disk storage goes through mmap and the OS page cache deliberately, making “RAM as a cache, not the dataset” a real, testable architectural pattern rather than a workaround. And critically for a filtered-search use case like ecommerce, Qdrant builds filtering into the HNSW graph itself, with ACORN available for the harder case of multiple high-cardinality filters; instead of treating filtering as a bolt-on post-processing step. Together, these make Qdrant less a place to store vectors and more a toolkit for deliberately trading off RAM, disk, latency, and recall.

Qdrant gives you three storage choices that matter here:

In-memory vectors. Everything sits in RAM. This is the fastest option, and it’s also the most memory-hungry. The disk is touched only for persistence.

Memory-mapped (on-disk) vectors. Qdrant always stores vectors in a memory-mapped file on disk; the question is whether they’re also pulled into RAM. mmap’d files aren’t loaded into RAM directly — they go through the OS page cache, so frequently accessed pages end up resident in memory even though the “official” storage location is disk. With enough RAM available, this can get close to in-memory performance because the page cache does the same job RAM would.

On-disk HNSW. Qdrant can also place the index itself on disk rather than just the vectors. This cuts RAM usage further, but graph traversal can now trigger disk I/O, so the speed of your underlying storage starts to matter in a way it didn’t before. Qdrant’s own guidance is to treat this as an aggressive RAM-saving move, not a default performance configuration.

That gives us the first important distinction for the rest of this piece: “on disk” doesn’t mean “never in memory.” mmap lets the operating system decide, dynamically, which parts of your dataset deserve to be cached — which is a very different mental model from a hard RAM/disk split.

3. The Memory Math Before Optimizing It

Now put actual numbers against the billion-vector scenario, at the dimensionality this benchmark actually uses (384d), across the representations Qdrant supports.

These are raw vector sizes only. They don’t include the HNSW graph, payload indexes, or payload data, all of which need their own RAM budget on top of this table. But the shape of the table is already the point: moving from float32 to binary is roughly a 32x reduction in the vector storage bill before you’ve touched indexing or filtering at all. That’s why compression becomes the first lever worth pulling, and it’s the subject of the next section.

4. Quantization: The Biggest Memory Lever

Qdrant’s current documentation describes four quantization methods: Scalar, Binary, Product, and TurboQuant, each sitting at a different point on the compression/accuracy trade-off curve.

The basic idea is the same across all of them: instead of storing every vector component as a 32-bit float, store a compressed approximation instead.

• Scalar Quantization compresses each float32 component down to an int8, a straightforward 4x compression with relatively small accuracy loss.

• Binary Quantization goes much further, reducing each component to one or two bits — up to 32x compression — and is especially effective on high-dimensional vectors with the right distribution. It also unlocks a large speed advantage, because binary representations let Qdrant use CPU instructions like XOR and popcount for distance calculations instead of full floating-point math.

• Product Quantization clusters chunks of each vector and represents them with centroid indices, pushing compression up to 64x — at the cost of slower, non-SIMD-friendly distance calculations and a bigger accuracy hit, best reserved for cases where minimizing memory is the overriding concern.

• TurboQuant is Qdrant’s newer option, aiming for up to 32x compression while holding recall better across a wider range of embedding models than classical binary quantization.

None of these are free; every one of them removes information that was present in the original float32 vector. Which raises the actual engineering question this section exists to ask: how much compression can you apply before recall becomes unacceptable? That’s an empirical question, not a documentation question, and it’s what Section 20 of this post actually measures.

One more thing worth knowing about here even before running numbers: Qdrant supports asymmetric quantization — storing vectors as binary but scoring incoming queries with scalar quantization instead. This keeps the RAM footprint close to pure binary while giving noticeably better precision, which matters most in memory-constrained or disk-I/O-bound deployments. It wasn’t part of this benchmark, but it’s a relevant knob if binary quantization’s recall loss turns out to be a problem for your workload.

5. Don’t Just Compress the Vectors — Think About the Retrieval Path

Compressing vectors is only half the story. The more interesting move is what Qdrant lets you do with the compressed and original vectors together.

Qdrant can store quantized vectors alongside the original, full-precision ones. That opens up a specific retrieval pattern:

Qdrant explicitly supports rescoring candidates against their original, uncompressed vectors after an initial quantized search. The idea behind this is worth stating plainly, because it reframes the whole compression question: you don’t need full precision for every comparison in the dataset. You need full precision only for the small number of candidates that might actually make it into the final result set.

That’s a very different cost structure than “compress everything and accept whatever recall you get.” It means the compression decision and the retrieval-path decision aren’t the same decision — you can be aggressive with quantization precisely because you’re not relying on it alone to produce the final ranking. Section 21 measures exactly how much recall this recovers, and it turns out to be substantial.

6. The Memory-Efficient Qdrant Configuration

This is deliberately different from just “quantize everything” or just “put everything on disk.” The quantized vectors; small, fast to scan; stay resident in RAM where the bulk of the search work happens. The original, full-precision vectors — large, but only needed for a handful of candidates per query — live on disk and get pulled in only when rescoring actually needs them.

The economics here is the whole point: keeping every full-precision vector in RAM for a billion-vector collection is the ~1.4 TB (at 384d) or ~3 TB (at 768d) number from Section 3. Keeping only the quantized vectors in RAM, with originals on disk and touched only for the rescoring step, is potentially an order of magnitude cheaper; without giving up the accuracy that rescoring recovers. Sections 19–21 test whether that promise holds up on the actual benchmark data, and where it starts to cost you.

7. mmap: Making Disk Part of the Memory Hierarchy

Section 2 introduced mmap in passing. It’s worth slowing down here, because it’s the mechanism that makes everything in Section 6 actually work.

With Qdrant’s mmap storage, the access path looks like this:

Application

↕

Page cache

↕

NVMe

The vector dataset doesn’t need to be resident in physical RAM to be usable. Instead, the operating system’s page cache holds whatever’s been accessed recently, and cold data gets pulled in from disk only when something actually asks for it. Qdrant’s own documentation states this plainly: with sufficient RAM, mmap-backed storage can get almost as fast as pure in-memory storage, because the page cache ends up doing the same job.

A few pieces make this work, and they’re worth naming individually because they show up again later in the filtering and HNSW-on-disk discussions:

• Virtual memory / mmap — the file on disk gets mapped into the process’s address space; reads and writes go through normal memory access instructions rather than explicit file I/O calls.

• Page cache — the OS keeps recently-touched pages of that mapped file in RAM automatically, without the application managing it.

• Page faults — when a query touches data that isn’t currently cached, the OS has to fault it in from disk, which is where the latency cost actually shows up.

• Hot vs. cold data — frequently queried vectors stay warm in the page cache; rarely touched ones get evicted and re-fetched from disk when needed again.

• NVMe speed — because page faults ultimately hit the disk, how fast that disk is directly bounds how bad a cold read can get.

The important nuance, stated as plainly as Qdrant states it: mmap doesn’t make disk access as fast as RAM. What it does is let you use RAM more selectively:spending it on the working set that actually gets queried, instead of the entire dataset regardless of access pattern. That’s a meaningfully different claim than “disk is basically free now,” and it’s worth testing rather than taking on faith — which is exactly what the mmap-specific benchmark later in this post does.

8. Three Configurations, Compared

Stack Sections 6 and 7 together and there isn’t just one “memory-efficient” setup; there’s a spectrum, and where you land on it is a real trade-off between RAM footprint and I/O exposure.

Now the graph itself is off RAM too. This is the most memory-frugal option, and also the one where I/O costs stop being theoretical.

Qdrant’s own documentation explicitly states that pushing both vectors and HNSW onto disk can cut RAM substantially, but graph traversal itself may now require I/O, which is a fundamentally different cost than the “only touch disk for rescoring” pattern in Configuration B. In this benchmark, these three map directly onto what got built and measured as Configs A, B/C, and D (Section 18 covers naming); the latency deltas between them, especially the jump from B to D, are one of the more concrete results this post has.

9. HNSW: The Other Major Memory Consumer

It’s tempting to treat vector compression as the whole memory story. It isn’t. HNSW :- the graph index Qdrant uses for dense-vector search, has its own memory footprint, and at a billion-vector scale, it stops being a rounding error.

The parameters that matter most:

• m — how many edges each node in the graph keeps. Higher m generally means better recall, at the cost of a bigger graph.

• ef_construct — how wide the search is during index construction. Higher values build a better-connected graph, at the cost of longer build time.

• ef — the equivalent search-width parameter at query time. Higher ef means more candidates get examined per query, trading latency for recall.

Every one of these knobs pushes graph size up or down independently of whatever you did with vector compression. That’s the point worth underlining here: compressing the vectors does not automatically solve the entire memory problem. A collection can have beautifully compressed, binary-quantized vectors and still be memory-heavy if the HNSW graph on top of it is large and fully RAM-resident. The two costs are separate line items, and treating them as one is how memory budgets end up wrong in practice.

10. HNSW on Disk: When Does It Make Sense?

Section 8’s Configuration C put the HNSW graph on disk alongside the original vectors. It’s worth pulling that apart on its own, because the trade-off is sharper than the vector-storage decision.

The benefit is straightforward: RAM requirements drop dramatically, since the graph, which can be a meaningful chunk of total memory at scale is no longer resident.

The cost is where it gets interesting. Graph traversal is inherently a sequence of “look at this node, then jump to its neighbors” operations, and when those neighbors aren’t in RAM, each jump can trigger disk I/O. That makes:

• Latency more dependent on storage speed than it was before,

• Random access patterns matter in a way they didn’t when the whole graph was RAM-resident (traversal doesn’t read sequentially, it hops).

This is exactly why Qdrant recommends fast NVMe specifically for this configuration, a graph traversal that has to wait on spinning disk or slow network storage for every hop is going to feel very different from the one backed by NVMe.

The central engineering question here isn’t “can I save RAM by doing this”; you obviously can. It’s: is the RAM saved worth the additional I/O it costs? That’s not answerable in the abstract; it depends on your latency budget and your storage. Config D in this benchmark is exactly this setup, and its latency numbers (Section 19 onward) give one concrete answer for one specific hardware/dataset combination, not a universal one.

11. Filtering Is Also a Scaling Optimization

Zoom out from pure vector storage for a moment. Real search queries in ecommerce are almost never “find things similar to this vector” in isolation; they come with structured constraints attached.

Take a realistic query: “Find running shoes similar to this product, under ₹10,000, size 9, in stock, and deliverable to Mumbai.”

The vector search here isn’t operating over the whole collection anymore. It’s implicitly scoped by:

category = running shoes

price <= 10,000

size = 9

inventory > 0

location = Mumbai

Qdrant’s payload indexes make this kind of filtering efficient, and more importantly for a billion-vector system, they can influence how the query planner approaches the search in the first place. That’s the framing worth carrying forward: filtering isn’t a separate concern bolted onto vector search. It changes the effective size of the search space, and therefore the actual cost of retrieval. A query with a highly selective filter is a fundamentally cheaper problem than one with no filter at all, if the engine is built to take advantage of that, which is exactly what the next two sections are about.

12. Qdrant’s Filterable HNSW

Here’s the problem with naïve filtering on top of an ANN graph: a sufficiently restrictive filter can wreck ordinary HNSW traversal, because many of a node’s graph neighbors simply won’t satisfy the filter. Follow enough dead-end edges and the search stalls before it ever reaches the true nearest matches — even though nothing is technically wrong with the graph.

Traditional (post-filter):

HNSW search → Candidates → Filter

Qdrant’s filter-aware approach:

HNSW traversal + Payload index → Filter-aware candidate traversal

Qdrant’s answer, documented as “filterable HNSW,” is to bake the fix into the graph itself rather than filtering after the fact. When a payload field is indexed, Qdrant walks its values and adds extra HNSW edges between points that share a value in that field — so a query filtered to that value still has a connected graph to traverse, instead of hitting a scattered set of islands.

This isn’t free. Those extra edges get added per indexed field, not per combination of fields, and they cost real build time. Qdrant’s own published benchmark on a one-million-point collection saw index build time go from about 116 seconds with no extra edges to 507–650 seconds with them, roughly 4.4x–5.6x longer. And there’s a size cap: a payload value shared by too many points (roughly a fifth of the collection or more, depending on graph density) gets skipped entirely, on the theory that the main graph should already keep that many points connected without help.

The framing that matters for the rest of this post: filtering → search efficiency → memory → billion-scale architecture are not four separate topics. They’re one connected decision, and filterable HNSW is Qdrant’s default answer to it.

13. High-Cardinality and Multiple Filters — Where ACORN Comes In

Push the query harder: “Find size-9 trail-running shoes under ₹10,000, in stock in Mumbai, from brands I’ve purchased before.” Now there are multiple filters stacked together, and that’s exactly where filterable HNSW’s per-field edges start to fall short — because those extra edges are built per field, not per combination, a two-filter intersection can land somewhere no single field’s edges actually cover.

This is where Qdrant’s ACORN mechanism comes in. Rather than repairing the graph at index time, ACORN repairs traversal at query time: when direct neighbors of a node have been filtered out, it looks one hop further, at neighbors of neighbors; instead of giving up. This recovers accuracy on exactly the cases filterable HNSW’s static edges miss, at the cost of extra work per query. It’s opt-in per query via the acorn search parameter, so enabling it doesn’t require rebuilding anything.

Qdrant’s own benchmark work on this (a separate one-million-vector test, not this project’s dataset) is a useful reference point for what ACORN actually buys you: on single-field filters, filterable HNSW alone already got very close to full recall for most selectivities, and ACORN mattered most specifically on the fields that got no extra edges — because their values were too common to qualify — and on two-field intersections, where extra edges from either individual field didn’t cover the combined constraint. On a 4% double-filter intersection, for instance, plain filterable HNSW recall dropped to the 60–70% range while ACORN recovered it close to 100%, at several times the latency. At very low selectivity (a fraction of a percent matching), Qdrant’s planner tends to skip the graph entirely and read straight from the payload index instead, which turned out to be the cheapest and most accurate path in that regime.

That last point is worth sitting with, because it previews something this project’s own filtering benchmark ran into directly: at high enough selectivity, or at small enough scale, the graph-repair mechanisms may simply have nothing to fix. Section 23 of this post covers what happened when this dataset’s own high-selectivity filter tier was tested against ACORN and the result wasn’t the clean “ACORN helps” story the documentation-level discussion might suggest.

The trade-off worth carrying forward either way: filter accuracy vs. search performance, and the right answer depends on filter selectivity, dataset scale, and how many strict filters get combined at once; not a fixed rule.

14. Payload Indexes Also Cost Memory

Filtering isn’t free just because it makes search faster. Qdrant’s payload indexes, the structures that make filterable HNSW and fast filter matching possible, consume their own memory and disk space, and Qdrant’s documentation is direct about the implication: index the fields you actually filter on, not everything in the payload.

That’s a genuine trade-off, not a formality:

More indexes

↓

Faster filtering

↓

More memory + disk

In practice this means someone has to decide, deliberately, which payload fields earn an index. A field nobody filters on is pure memory cost with zero retrieval benefit — and at scale, “index everything just in case” is exactly the kind of decision that quietly erodes all the RAM savings won earlier from quantization and mmap.

This is a real decision this project had to make, not just a documentation point. The raw H&M dataset carries 33 columns. Only 8 of them — product_id, title, description, category, brand, price, availability, geography — are actually used for filtering or display, and only those made it into the payload. The other 25 columns exist in the source data but aren’t stored in Qdrant at all, precisely because indexing or storing fields nobody queries costs memory for no retrieval benefit.

15. Query Planning: Sometimes Full Scan Beats HNSW

Here’s a detail that runs against the usual instinct at a billion-vector scale, where “just use approximate nearest neighbor search” feels like the obvious answer.

Qdrant’s query planner can choose to skip HNSW entirely and fall back to a full scan, when the filtered subset of the collection is estimated to be small enough. This is decided per query, based on the estimated size of the data satisfying the filter condition, against a configurable threshold (full_scan_threshold, measured in kilobytes of vector data).

Why this makes sense once you think about it: ANN search exists to avoid scanning the entire dataset. If a filter has already cut the candidate set down to a few hundred or a few thousand vectors, scanning those directly can be cheaper than traversing a graph structure built to handle a search space many orders of magnitude larger. The graph’s overhead, hopping between nodes, checking edges doesn’t pay for itself once there’s barely anything left to search.

This is also why “use HNSW because you have a billion vectors” is an incomplete rule. The real rule is closer to: use HNSW when the effective search space is large, and let the planner fall back to brute force when a filter has already done most of the work for you. It’s a small detail, but it’s the kind of thing that separates a system that’s just running ANN everywhere from one that’s actually reasoning about the query it’s serving.

16. RAM Is a Cache, Not Necessarily the Dataset

Pull Sections 2–15 together and a single mental model falls out of all of it.

The goal was never to fit the entire database into RAM. The goal is to fit the right working set into RAM — the quantized vectors that carry most of the search load, the hot HNSW structures, the payload indexes actually used for filtering — and let mmap and the OS page cache handle everything colder.

Qdrant’s own capacity-planning guidance makes this explicit: frequently accessed data should stay in memory, and the rest can be offloaded to disk without the system falling over. It’s a reframe worth stating plainly, because it’s the thesis the entire first half of this post has been building toward: billion-vector search isn’t a RAM-sizing problem. It’s a working-set problem.

That’s the last of the conceptual groundwork. From here, the post moves into what actually got built and measured.

17. The Dataset and Query Set

Time to move from architecture to what was actually built and measured. You can refer to the code and scripts here.

Github

The benchmark runs on 105,126 real products from Qdrant’s own published Qdrant/hm_ecommerce_products dataset — real H&M titles, descriptions, categories, and precomputed 384-dimension BGE-small embeddings. This was a deliberate choice over generating fully synthetic data: real product text and real embeddings produce a realistic similarity structure that synthetic data can’t easily fake, and the dataset was already sized and formatted for exactly this kind of benchmark.

Three fields the schema needs — price, availability, and geography — don’t exist in any real-world source for this catalog, so those were generated synthetically and layered on top. Everything else (title, description, category, brand, embedding) is the genuine dataset.

Two schema decisions are worth stating explicitly, because they shaped everything downstream:

Dimensionality stayed at 384d. Re-embedding at a higher dimension was considered, to match a commonly-cited illustrative figure, but rejected — that figure was never a real requirement, and re-embedding 105K products would have thrown away the real BGE-small embeddings for no actual benefit. All the memory math in this post (Section 3 onward) is computed at the real 384d. Price, availability, and geography live in the payload, not the vector. They’re structured filter metadata, not semantic content, putting them in the vector would conflate “similar in meaning” with “similar in price,” which isn’t the retrieval behavior anyone actually wants. The embedding column itself needed a correction along the way: the dataset’s readme documents the column as bge_embedding, but the actual column in the live dataset is dense_embedding. This is confirmed against the HuggingFace dataset viewer and fixed before any collection was built; a small thing, but the kind of detail that silently breaks an entire pipeline if it’s caught late.

The payload was narrowed from the dataset’s 33 raw columns down to the 8 actually used for filtering or display, per the memory reasoning in Section 14: product_id, title, description, category, brand, price, availability, geography.

Query resolution had its own bug worth flagging here since it affects every result downstream: the first attempt resolved queries by matching on title, but many H&M products share a title across different colorways or sizes so the same title could silently map to different products between runs, producing inconsistent query counts run to run. This is fixed by joining on product_id, which is guaranteed unique, both when generating queries and when resolving them for benchmarking. Every recall and latency number in this post reflects that fix.

18. Experimental Configurations

Five configurations were built, in strict order, changing one variable at a time — the same discipline the earlier architecture sections argued for, applied for real:

This maps directly onto the three-configuration comparison from Section 8: A is “everything in RAM,” B/C is the quantized-in-RAM-with-originals-on-disk hybrid from Section 6, and D is the aggressive disk-backed setup from Section 10. Building them in this exact order — one variable per step — is what makes the results in the next few sections actually comparable to each other, instead of conflating “quantization changed things” with “storage location also changed things” in the same measurement.

Two indexing bugs surfaced during this build phase and are worth stating here because they explain why the numbers in this post are trustworthy rather than just plausible-looking:

First, indexed_vectors_count got stuck at 0 out of 105,126 on the first attempt. The cause was a misunderstanding of indexing_threshold=0 — it was assumed this would force immediate indexing, but it actually disables indexing entirely. This was fixed by removing it at collection-creation time and instead lowering the threshold to a small non-zero value (1,000) after upload.

Second, and more consequential: before that fix, roughly 6,000 leftover points per collection stayed unindexed and were quietly brute-force searched alongside the indexed ones — which inflates recall and corrupts latency, since exact search and approximate search were getting blended into a single reported number without anyone intending that. This was fixed by adding a wait_for_indexing() step that polls until indexed_vectors_count matches points_count and status is green, before any benchmark is allowed to run. Every collection below reports indexed_vectors_count: 105126 — fully indexed, not partially.

19. Quantization Results

With Configs A and B fully indexed, here’s what full precision vs. scalar vs. binary quantization actually looked like, at 100 queries per collection:

A couple of things worth pausing on here, because they’re not quite what the documentation-level story predicts.

Recall@10 for BQ came in higher than SQ (0.978 vs. 0.928). Section 4 framed binary quantization as the lossier of the two, and it generally is — but that comparison here is doing something specific: rescore=False was passed explicitly for both collections in this table, and BQ’s recall gap tends to show up specifically without rescoring, on the tail (Recall@100, where BQ drops to 0.9264 against SQ’s 0.9566). At Recall@10, on this particular embedding model and query set, binary quantization actually held up better than expected. That’s a real, measured result;not the outcome either method’s compression ratio alone would predict, and a good example of why “just check the docs” isn’t a substitute for actually running the benchmark on your own data.

The rescore=False finding is worth its own paragraph, because it changed what “no rescoring” actually meant in this benchmark. Early on, quantization_params=None (i.e., not setting the parameter at all) was assumed to be equivalent to explicitly setting rescore=False. It isn’t. On the identical BQ collection, leaving the parameter unset produced a materially better recall number than explicitly disabling rescoring — a gap between 0.742 and 0.978 recall@10 depending on which of the two you did. Left implicit, the benchmark had been silently reporting a much-better-than-real “no rescoring” number for BQ in an earlier pass. The table above uses the corrected, explicit rescore=False — forced for quantized collections directly in the benchmark CLI — so 0.978 recall@10 for BQ is the honest number, not the inflated one.

Latency roughly doubled for both quantized collections versus baseline (3.08ms → ~7.8ms p50). This runs against the usual pitch for quantization — smaller vectors, faster comparisons — and it’s worth being straight about it rather than smoothing it over: at this specific scale (105K vectors, single machine), the quantized collections were slower, not faster, on raw latency. This is almost certainly an artifact of scale and machine-level noise rather than a real property of quantization — binary quantization’s whole performance case rests on cheap bitwise operations at scale that a 105K-vector collection may simply not exercise enough to show. It’s flagged here rather than explained away, because a benchmark that only reports the numbers that match the documentation’s story isn’t a real benchmark.

The core signal does hold up cleanly: scalar quantization gives roughly 4x compression with a moderate accuracy cost, and binary quantization compresses further with a real accuracy trade-off that,as the next section shows,mostly disappears once rescoring is turned back on.

20. Rescoring Results

Section 5 made the case, at the documentation level, that you don’t need full precision for every comparison,just for the small set of candidates that might make the final result. This is where that claim gets tested against real numbers.

The sweep varies oversampling — how many extra candidates get pulled before rescoring trims back down to the final top-K — from no rescoring at all up to 8x, on both SQ and BQ collections:

**Scalar quantization:**

| Setting | Recall@10 | Recall@100 | p50 | p95 | p99 |
| ----- | ----- | ----- | ----- | ----- | ----- |
| No rescoring | 0.928 | 0.9566 | 2.65 ms | 4.86 ms | 24.48 ms |
| Rescore, 1x | 0.990 | 0.9996 | 2.65 ms | 3.98 ms | 4.34 ms |
| Rescore, 2x | 0.990 | 0.9998 | 2.46 ms | 3.60 ms | 5.13 ms |
| Rescore, 4x | 0.988 | 0.9998 | 2.91 ms | 4.15 ms | 4.47 ms |
| Rescore, 8x | 0.988 | 0.9994 | 3.54 ms | 4.92 ms | 5.11 ms |

**Binary quantization:**

| Setting | Recall@10 | Recall@100 | p50 | p95 | p99 |
| ----- | ----- | ----- | ----- | ----- | ----- |
| No rescoring | 0.742 | 0.6448 | 4.65 ms | 6.43 ms | 10.41 ms |
| Rescore, 1x | 0.978 | 0.9264 | 2.29 ms | 3.36 ms | 3.80 ms |
| Rescore, 2x | 0.982 | 0.9776 | 2.43 ms | 3.22 ms | 3.71 ms |
| Rescore, 4x | 0.984 | 0.9942 | 2.39 ms | 3.65 ms | 5.17 ms |
| Rescore, 8x | 0.988 | 0.9986 | 3.30 ms | 4.45 ms | 5.10 ms |

The headline number: BQ recall@10 goes from 0.742 to 0.978 with just 1x oversampling turned on — a single extra pass of candidates rescored against the original vectors recovers almost all of binary quantization’s accuracy loss. That’s the core claim from Section 5 holding up under an actual measurement, not just a plausible architecture diagram.

Two things worth noting beyond the headline number:

Returns diminish fast. Recall@10 for BQ barely moves between 2x and 8x oversampling (0.982 → 0.988), while Recall@100 keeps climbing more noticeably (0.9776 → 0.9986) — rescoring more candidates mostly helps the long tail of results, not the top handful, which already gets fixed almost immediately.

Rescoring didn’t just fix recall; it made latency better, not worse. This is the part that runs against the naive intuition that rescoring is a “pay accuracy back with latency” trade. Compare BQ’s no-rescoring p99 latency (10.41ms) against 1x rescoring’s p99 (3.80ms): rescoring is faster on the tail, not slower. The likely explanation is that without rescoring, some queries were falling back to more expensive comparison paths internally; with a small, tightly bounded rescoring step against a compact candidate set, the tail gets more predictable rather than less. Either way, on this dataset there’s essentially no reason to skip rescoring — it improved both recall and worst-case latency simultaneously.

Put together with Section 4’s framing: this is the practical version of “you don’t need full precision for every comparison.” A 1x-oversampled rescore against the original vectors — touching only a small multiple of the final result count, not the whole candidate pool — recovered nearly all of binary quantization’s lost accuracy, at no latency cost. That’s the actual case for the quantize-then-rescore architecture from Section 6, measured rather than asserted.

21. mmap and On-Disk Results (Configs C and D)

Now let’s come to the storage-location question from Sections 6–10, measured rather than argued. All numbers below are without rescoring, same 100-query set, same fully-indexed collections.

**Scalar quantization, across storage configs:**

| Config | Recall@10 | p50 | p95 | p99 |
| ----- | ----- | ----- | ----- | ----- |
| B — RAM only | 0.928 | 7.77 ms | 11.69 ms | 20.93 ms |
| C — originals on disk | 0.914 | 5.94 ms | 11.35 ms | 30.13 ms |
| D — originals \+ HNSW on disk | 0.920 | 6.15 ms | 10.20 ms | 25.69 ms |

**Binary quantization, across storage configs:**

| Config | Recall@10 | p50 | p95 | p99 |
| ----- | ----- | ----- | ----- | ----- |
| B — RAM only | 0.978 | 7.81 ms | 11.81 ms | 19.33 ms |
| C — originals on disk | 0.982 | 14.83 ms | 23.87 ms | 47.14 ms |
| D — originals \+ HNSW on disk | 0.984 | 2.45 ms | 4.22 ms | 6.72 ms |

Two very different stories in these two tables, and both deserve honesty rather than a tidy narrative.

Scalar quantization roughly matches what Section 7’s mmap theory predicts. Recall stays essentially flat across B, C, and D (0.928 → 0.914 → 0.920 — within noise for a 100-query sample). Median latency (p50) doesn’t get meaningfully worse moving vectors and even HNSW to disk — consistent with the session’s own read of this result: without rescoring, search never actually touches the original vectors, so putting them on disk is largely invisible at the median. Where the disk cost does show up is the tail: p99 climbs from 20.93ms (B) to 30.13ms ( C )once original vectors move to disk, which lines up with Section 7’s page-fault story — most queries hit the warm page cache, but the unlucky ones pay a real disk-read cost.

Binary quantization tells a much messier story, and it’s worth being upfront about why. The BQ-on-disk row got noticeably slower — p50 nearly doubling to 14.83ms — while the BQ-disk-plus-HNSW-disk row (D) came back faster than the RAM-only baseline (2.45ms vs. 7.81ms p50). That second result — disk-backed HNSW outperforming RAM-only HNSW — isn’t architecturally plausible on its face, and it wasn’t treated as a real finding. It’s the specific symptom of a benchmarking bug caught during this project: back-to-back runs were biasing latency numbers through the OS page cache, because whichever collection got benchmarked most recently (or most repeatedly) inherited a warmer cache than the ones benchmarked before it. The fix was adding an untimed warm-up pass ahead of every timed run, specifically to prevent one collection’s numbers from riding on another’s residual cache state. Rather than quietly re-running until the numbers looked reasonable, this result is flagged here as what it is: a measurement artifact, not a claim that disk-backed HNSW beats RAM.

The honest takeaway from this pair of tables: the mmap claim from Section 7 holds up for scalar quantization at this scale — flat recall, tail-latency cost from disk reads, nothing more dramatic. The binary quantization numbers in this configuration need to be treated with more caution and, ideally, rerun cleanly with the warm-up fix applied consistently before being trusted for anything beyond “something about cache-state ordering was going on here.”

22. A Note on Storage-Class Comparisons

Section 10 raised the natural follow-up question: at what point does storage latency actually overwhelm the benefit of putting HNSW on disk? The honest answer here is that this benchmark can’t say — it ran on a single machine with one storage tier, so there was no NVMe-vs-slower-SSD-vs-network-storage comparison to run. Section 21’s Config D numbers show what one specific storage class looks like; they don’t say anything about where the crossover point is against something slower. That’s a real open question this project didn’t have the infrastructure to answer, not a claim being quietly skipped over.

23. Filtering and ACORN Results

Section 13 set up the promise: filterable HNSW handles most filter shapes well, and ACORN is supposed to recover accuracy on the shapes it doesn’t — high-cardinality, multi-filter, high-selectivity queries. Here’s what actually happened when that was tested on this dataset.

Filters were built across selectivity tiers, from broad (L1) down to highly restrictive (L3, and later an even tighter L4):

• L1/L2 (broad to medium filters) showed no recall cost at all versus unfiltered search. This matches Section 12’s story cleanly — filterable HNSW’s extra edges are doing their job at these selectivities, and there’s nothing for ACORN to fix.

• L3 (high selectivity, roughly 0.47% of the corpus after a correction described below) showed no measurable ACORN benefit — recall was already at 1.0 without it. In other words, at this tier there was nothing broken for ACORN to repair.

That L3 result runs directly against the intuition Section 13 built up from Qdrant’s own published ACORN benchmark, where high-selectivity and multi-filter combinations were exactly where ACORN earned its keep. The likely explanation isn’t that ACORN doesn’t work,it’s scale. Qdrant’s own published ACORN benchmark shows the effect clearly at roughly 5 million vectors and ~4% selectivity. This benchmark runs at ~105K vectors. At that much smaller scale, filterable HNSW’s index-time edges may already preserve enough connectivity that there’s simply no broken graph for ACORN’s second-hop traversal to repair — the effect Section 13’s own referenced benchmark shows may genuinely require more vectors, or a tighter selectivity, than this dataset provides. An L4 “extreme selectivity” tier (tighter price headroom plus a fifth filter field) was added specifically to test whether the ACORN effect would show up at all in this dataset before concluding it’s simply out of reach at this scale — that result was not yet reviewed at the time of writing, and is left here as an open item rather than a filled-in number.

One filtering bug is worth including here as its own finding, because it’s a good example of how a benchmark can look reasonable while quietly measuring the wrong thing. The first attempt at building high-selectivity filters sampled category, brand, and price fields independently at random. That produced a filter tier where, on average, only 0.2 matching candidates existed out of 105,126 — because category and brand aren’t actually independent in a real product catalog, so random independent sampling mostly generated filter combinations that don’t correspond to any real product at all. The fix was to derive each filter from an actual product row’s own category, brand, and price (with some headroom on price), which guarantees at least one real match while staying genuinely restrictive. The 0.47% L3 selectivity figure above reflects that fix — the earlier, broken version of this filter tier would have produced numbers that looked like a system failure when it was actually a bad test-data generator.

Also worth stating plainly rather than glossing over: at this dataset’s scale, absolute latency for filtered queries is often sub-2ms, which is likely below the noise floor for reliably distinguishing ACORN’s per-query overhead from ordinary measurement jitter. That’s a scale limitation worth naming explicitly — it’s not something more query repetitions alone would fix, since the signal being measured may simply be smaller than the noise at this collection size.

24. The Final Architecture

Pulling everything from Sections 2–23 together, here’s the architecture this benchmark actually supports, not as a universal prescription, but as one reasonable answer to the trade-offs measured above:

This is the shape that Section 20’s rescoring numbers and Section 21’s scalar-quantization mmap numbers actually justify: quantized vectors and the hot parts of the index stay in RAM because that’s where the bulk of search cost lives; original vectors sit on disk because rescoring only needs to touch a small, bounded set of them per query, not the whole collection.

Worth restating plainly: this is a reasonable architecture given what this benchmark measured, on this dataset, at this scale — not the correct architecture for every deployment. A workload with a much tighter latency budget, or a much larger collection where the disk-backed HNSW question in Section 21 actually gets tested cleanly, could reasonably land somewhere else on this same spectrum.

25. When Should You Put HNSW on Disk?

A practical decision framework, grounded in what Section 21 actually showed rather than stated as a rule of thumb:

Keep HNSW in RAM when:

• Latency is critical — Config B’s ~7.8ms p50 (SQ and BQ alike) is the RAM-resident baseline to bea

• RAM is available and the collection doesn’t force the issue

• The workload is large and highly concurrent, where disk-backed graph traversal under load compounds badly.

Put HNSW on disk when:

• RAM is severely constrained relative to collection size

• Dataset size dominates infrastructure cost more than latency does

• NVMe is genuinely fast — Section 22 is a reminder that this framework assumes fast local storage, not something slower

• Somewhat higher tail latency is acceptable — Section 21’s scalar-quantization result (p99 climbing from ~21ms to ~30ms moving to disk) is a realistic size for that cost, at this scale

Qdrant’s own guidelines treat putting both vectors and HNSW on disk as a more aggressive RAM-saving configuration, not a default performance posture — and the measured numbers here back that framing up. The scalar quantization results support the “this is workable” case cleanly. The binary quantization results from the same config were compromised by the cache-ordering bug described in Section 21, so — in the spirit of not overstating what was actually shown — the honest answer for BQ specifically on disk-backed HNSW is “re-run this cleanly before trusting a number,” not a confident recommendation either way.

26. When Should You Use Quantization?

Section 4 laid out what each method is documented to do. Here’s how that guidance holds up against what actually got measured:

Scalar quantization — a good starting point when you want substantial compression with a comparatively small accuracy hit. That held up: ~4x compression, recall@10 of 0.928 without rescoring, and 0.99 with even 1x oversampling.

Binary quantization — useful when memory reduction matters more than anything else, particularly on suitable high-dimensional distributions. Also held up, with a caveat worth repeating: without rescoring, BQ’s Recall@100 in particular took a real hit (0.6448) — this is not a method to run without rescoring turned on, based on what Section 20 showed. With 1x rescoring, the accuracy case becomes very strong.

Product quantization — documented as useful when minimizing memory is the overriding concern, at the cost of more complexity and a bigger accuracy trade-off. Not built in this project — it was explicitly optional given the time available, and there’s no measured data here to report on it one way or the other.

TurboQuant — worth evaluating when the embedding model and quality requirements make its higher compression attractive. Also not built here, for the same reason.

The one method this project can’t speak to from direct measurement is exactly the one Qdrant’s documentation frames as the “if minimizing memory is the overriding concern” option — which is a fair gap to flag rather than paper over with confident-sounding guidance.

27. The Real Optimization Target

It’s tempting, after all of the above, to summarize this whole post as “minimize RAM.” That’s incomplete, and worth rejecting explicitly. A system running 200GB of RAM against very slow NVMe can genuinely be worse than one running 500GB against fast storage — Section 21’s disk-latency numbers are a small-scale preview of exactly that trade-off. Similarly, a system chasing 99% recall at enormous infrastructure cost can be a worse decision than one settling for 98.5% recall at a fraction of the cost — which is essentially the choice Section 19 and 20 hand you directly: BQ with 1x rescoring gets extremely close to full-precision recall at a fraction of the RAM footprint.

The actual optimization target is:

Cost ↔ RAM ↔ storage ↔ latency ↔ recall ↔ throughput

One honest gap to flag here: this project didn’t get to computing the cost side of that equation directly — translating the RAM and disk footprints measured above into an actual $/GB comparison across configurations. That’s a natural next step on top of everything measured so far, not something this benchmark set out to answer.

28. Conclusion

Go back to where this started: a billion product embeddings, and the instinct to just throw everything in RAM.

The answer this post actually supports isn’t a single trick. It’s a hierarchy, and each piece of it did something specific and measurable:

HNSW → avoid exhaustive search over the whole collection.
Filtering → reduce the effective search space before vector search even runs.
Quantization → make the vector representation dramatically smaller — 4x with scalar, more with binary — measured directly in Section 19.
mmap → let the dataset exceed physical RAM, with the OS page cache doing the work of deciding what stays hot — confirmed for scalar quantization in Section 21.
RAM-resident quantized vectors → keep the representation that carries most of the search load fast and close.
Original vectors on disk → retain full-precision data economically, touched only when rescoring actually needs it.
Rescoring → recover the accuracy approximate retrieval gave up — and, on this dataset, at essentially no latency cost, which was the most concrete positive surprise in the entire benchmark.
Disk-backed HNSW, when it’s worth it → push memory requirements lower still, accepting the I/O cost that comes with it — a trade-off this benchmark could measure but not fully validate at this scale.

At a billion-vector scale, the question was never really “how do I fit everything into RAM.” It’s “which parts of my retrieval pipeline actually deserve RAM, and which can be efficiently backed by storage” — and that’s not a question with one universal answer. It’s a question a real benchmark, run on real data, has to actually answer for your specific dataset, filters, and latency budget. This one did that at 105K vectors. The next honest step is doing it again at a scale where the disk-backed HNSW and ACORN questions this post had to leave open finally get resolved.

First published on medium

References

• Qdrant Documentation — Storage: https://qdrant.tech/documentation/concepts/storage/

• Qdrant Documentation — Quantization: https://qdrant.tech/documentation/guides/quantization/

• Qdrant Documentation — Indexing (filterable HNSW, ACORN, payload indexes, query planning): https://qdrant.tech/documentation/concepts/indexing/

• Qdrant Documentation — Search (ACORN parameter, query planning): https://qdrant.tech/documentation/concepts/search/

• Qdrant Documentation — Capacity Planning: https://qdrant.tech/documentation/guides/capacity-planning/

• Qdrant Blog — “Filtered Vector Search: What ACORN Fixes, and What Fixes ACORN”: https://qdrant.tech/articles/filtered-vector-search-acorn/

• Qdrant Blog — “Pre-Filtering vs Post-Filtering (and Why Qdrant Does Neither)”: https://qdrant.tech/blog/pre-filtering-vs-post-filtering/

• Qdrant Article — “Minimal RAM to Serve 1M Vectors”: https://qdrant.tech/articles/memory-consumption/

• Qdrant Article — “Product Quantization for Vector Search”: https://qdrant.tech/articles/product-quantization/

• Dataset — Qdrant/hm_ecommerce_products (Hugging Face)

By the end of this guide, you will have an end-to-end agent running completely on your machine. It will allow you to pass a sentence like “that warm golden hour photo from the trip where we stopped at the dhaba” and instantly pull up the image from your gallery in under 100 ms—all while consuming less than 1 GB of RAM and exactly zero dollars.

You can find a fully functional GitHub repo to replicate the system below.

To learn more, follow along!

GitHub: https://github.com/vatsala-singh/AI-Powered-Photo-Search-and-Tagging-Agent

Introduction

If you are someone like me with a phone gallery filled with thousands of pictures from everywhere, you would understand the struggle of finding the right picture on time. Every time you want to find something important, like a picture of an event poster you spotted while walking, or the recipe of your favourite pasta that your friend scrawled on a napkin, rotting away somewhere in the gallery waiting to be found, or that payment screenshot of an order you placed a month ago, the struggle is very real.

And it is not just personal chaos. If you keenly observe, today I wouldn’t be wrong in saying that photos have become our primary mode of content consumption. Pick any social media platform, including LinkedIn, and the trends clearly gravitate toward the visual. Photos are no longer just memories. They are documents, receipts, notes, and sometimes the only record of something important.

I tried everything. Creating well-organised albums, segregating pictures into multiple buckets, naming folders like my life depended on it. Nothing really worked. Even though some phones now have a gallery search feature, the brutal truth is they only recognise a handful of keywords and have no real understanding of what is actually in the image. Ask it to find “that rainy evening at the chai stall” and watch it stare back blankly.

So I decided to stop waiting for someone else to solve this and do what most developers eventually do: build it myself.

That is where the journey began. I started brainstorming with my entire development team (ChatGPT, Gemini, Claude… you know how it is 😄), and after spending hours refining what I actually wanted, the idea became clear: a multimodal semantic search engine for my personal photo library.

The concept was straightforward enough. Take a multimodal model that understands both image and language, use it to generate vector embeddings that capture the meaning of each photo, store those embeddings in a database, and then whenever you want to search, simply fire a natural language query. The system embeds the query the same way, computes similarity against all the stored image vectors, and returns the most relevant photos — ranked by how closely they match what you are looking for.

But I had some absolute non-negotiables before I started:

• Zero-dollar overhead: The entire setup had to be completely free of cost. No recurring API fees, no subscription traps.

• Absolute privacy: My photos are personal. That meant no cloud services, no remote databases, and no third-party APIs should ever touch them. Everything had to happen on local iron.

• Ultra-lightweight footprint: It needed to run comfortably on an edge device—my everyday laptop, a mini PC, or eventually even a Raspberry Pi tucked away on a shelf.

• Zero lag: Despite relying entirely on local models, the retrieval had to feel instant. No one wants to wait ten seconds for a search bar to wake up.

And to be very honest, finding tools that checked all four boxes at the same time was not at all easy. It took weeks of exploring, dead ends, and a few setups that worked beautifully on paper but terribly in practice.

But this past weekend, I finally sat down and built something that fits just right.

And if you have been following my blogs, you already know, whenever I find something worth sharing, I do. Not just the final result, but the thought process, the wrong turns, the little moments where things clicked. Because the best part of building in public is that it opens a conversation, and some of the most useful things I have learned have come from fellow developers jumping in with their own experiences.

So with that, let us get into it. Let us talk about how to make the best out of the most underutilised pile of unstructured data most of us are quietly drowning in: our photo galleries.

Why Traditional Photo Search Is Broken

Before we get into what I built, it is worth spending a moment on why the existing solutions fall short. Because this is not an obvious problem, most people do not even realise how broken their photo search is until they desperately need to find something and cannot.

The album approach collapses in real life

The idea of organising photos into albums sounds perfectly reasonable. You create folders - Travel, Food, Work, Random, and file things away neatly. This works for about three weeks. Then life gets busy, photos pile up faster than you can sort them, and before long you have a 600-photo "Miscellaneous" dump that defeats the entire purpose.

Even when people do maintain albums consistently, albums only work if you know in advance what you will be looking for later. The event poster you photographed in passing? You were not thinking "I should file this under Street Signs." You just snapped it and moved on. That is how real life works.

Keyword search only gets you so far

Some gallery apps and a few phones have started offering keyword-based search. Type "beach" and it surfaces photos tagged or identified as beaches. This is genuinely useful, up to a point.

The problem is that keyword search is brittle. It depends entirely on whether the system knows the right word for what is in your image. Ask for "sunset" and it might work. Ask for "that warm golden hour photo from the trip where we stopped at the dhaba on the highway" and it has absolutely nothing to offer you. Keywords are labels, they are not understanding.

Cloud gallery search: good, but at a cost

Google Photos and Apple Photos have made real progress here. Their on-device and cloud models can recognise faces, scenes, objects, and even some text within images. For a lot of people, this is good enough. But good enough is not the same as the best choice. There are two fundamental problems with relying on cloud-powered gallery search.

First, your photos are being processed and indexed on someone else's servers. For most personal media, candid moments, private conversations captured in screenshots, medical documents photographed for reference, that is a meaningful privacy trade-off that deserves more thought than most people give it.

Second, these systems are closed. You cannot extend them, query them in new ways, plug them into your own workflows, or build anything on top of them. You are a user of their product, not the owner of your own data pipeline.

The real gap: no semantic understanding

Here is what all of these approaches have in common: they are all working at the level of labels and keywords. None of them are truly understanding the content of an image the way a person would. When I look at a photo, I do not think "outdoor, daytime, food." I think "oh, that is from the time we found that tiny Tibetan place after getting completely lost in Coorg." That memory is rich, contextual, and semantic. Current search tools have no way to connect a natural language description like that to the actual pixel content of an image. That is the gap we are going to close.

The Concept: Semantic Search, Explained Simply

Before we touch any code, I want to make sure the core idea is completely clear, because once it clicks, everything else in this build will feel obvious rather than magical. The central question we are trying to answer is: How do you teach a computer to understand what a photo is about, well enough that you can describe it in plain English and have the right one surface up? The answer lies in something called embeddings. So let’s understand it.

What is an embedding?

Think about how you would describe two photos to a friend. A picture of a golden retriever playing in a park and a picture of a labrador chasing a frisbee on a beach. Even though these are completely different images, pixel for pixel, your brain immediately recognises that they are similar. They share meaning. Both have dogs, both have open outdoor spaces, both feel energetic and playful.

Now imagine you could represent the meaning of any image as a point in space. Photos that are semantically similar: dogs at play, sunsets over water, receipts on a desk would cluster close together. Photos that are completely unrelated would sit far apart. That space is what we call a vector space, and the point that represents each image is its embedding.

An embedding is just a list of numbers – a few hundred to a few thousand of them – that together encode what an image means, not just what it looks like at the pixel level. Two images with similar embeddings are semantically similar, even if they share zero pixels in common.

Where does language come in?

The main challenge here is that of multimodality: we need something that can understand both image and text, and not just that, but also a model that can correlate the text to image. Enters CLIP - Contrastive Language Image Pre-Training, developed by OpenAI to understand image and text in the same vector space. What this means in practice is that if you feed CLIP an image of a dog on a beach, and separately feed it the text "dog playing near water", both produce embeddings that land close to each other in that shared 512-dimensional space.

This is the key insight that makes our entire system possible. We do not need to tag images manually. We do not need to match file names or keywords. We embed images once, embed a search query at query time, and find which images are closest to that query in the vector space.

Choosing the Right Tools (And Why It Took Weeks)

Now that the concept is clear, let me walk you through the tool choices. And I want to be honest here – this was not a clean, decisive process. It was weeks of reading documentation, running experiments, hitting memory limits, discovering that something "lightweight" still needed a GPU, and generally learning things the hard way. Here is what I was optimising for, and why it kept ruling things out:

Free: Ruled out hosted APIs and managed cloud services immediately

Private: Ruled out anything that needed to phone home, sync to the cloud, or send my photos anywhere

Lightweight: Ruled out models requiring a GPU or more than a few GB of RAM just to idle

Fast: Ruled out approaches where a single search query took several seconds With those four filters applied together, the field narrowed quickly. Here is what made the cut and why.

CLIP ViT-B/32 via FastEmbed — the eyes of the system

The embedding model is the heart of the whole setup, so this decision carried the most weight.

CLIP (specifically the ViT-B/32 variant) was the right call for a few reasons. First, it was literally designed for this, trained on hundreds of millions of image-text pairs specifically to align visual and linguistic meaning into one shared space. Second, the ViT-B/32 variant is genuinely lightweight: it runs comfortably on CPU, does not need a GPU, and the model weights are only around 150MB each for the image and text towers.

But raw CLIP through Hugging Face Transformers, while perfectly fine, requires you to handle preprocessing, batching, normalisation, and device management yourself. Enter FastEmbed. FastEmbed is a lightweight embedding library built by Qdrant, optimised specifically for CPU-friendly inference. It wraps CLIP under the hood, handles all the preprocessing details, auto-caches models locally, and makes embedding an image or a query a single function call. The two model variants we use:

Qdrant/clip-ViT-B-32-vision — image encoder, produces 512-d vectors

Qdrant/clip-ViT-B-32-text — text encoder, produces 512-d vectors in the same space

Both are loaded once when the server starts, cached in memory, and reused for every subsequent request; no reload penalty per query.

Architectural Note on ONNX vs. PyTorch: You’ll notice we are using FastEmbed rather than raw PyTorch via Hugging Face. FastEmbed utilizes an ONNX runtime under the hood. Because it strips out heavy training-related graphical dependencies, it maximizes CPU throughput, dropping our generation latency to a crisp 10–50ms per image.

from fastembed import ImageEmbeddingModel, TextEmbeddingModel

# Loaded once, reused forever
image_model = ImageEmbeddingModel.from_pretrained("Qdrant/clip-ViT-B-32-vision")
text_model = TextEmbeddingModel.from_pretrained("Qdrant/clip-ViT-B-32-text")

In practice: ~10–50ms per image on a modern laptop CPU, which is fast enough to batch-index thousands of photos in a single sitting.

# After embedding: 
image_vector = embed_image("sunset_photo.jpg") # shape: (512,) 
query_vector = embed_text("beautiful sunset") # shape: (512,) 
# Cosine similarity — just a dot product on normalised vectors 
similarity = np.dot(image_vector, query_vector) 
# If image is a sunset → similarity ≈ 0.85 (strong match) 
# If image is food → similarity ≈ 0.15 (weak match)

Qdrant Edge — the memory

Once you have embeddings, you need some place to store and search them efficiently. A regular database finds things by exact match or range filter; it has no concept of "closeness." A vector database is built specifically for nearest-neighbour search over high-dimensional embeddings, which is exactly what we need.

There are several good vector databases out there — Pinecone, Weaviate, Chroma, Milvus — but most are either cloud-hosted, resource-heavy, or require a running server process you have to manage separately.

Qdrant Edge is different in one very important way: it runs entirely in-process, embedded directly inside your Python application.

No server to spin up. No port to manage. No network overhead. You point it at a local folder, and it persists everything to disk automatically.

Your data never leaves your machine.

from qdrant_edge import(
    Distance,
    EdgeConfig,
    EdgeShard,
    EdgeVectorParams,
)
config = EdgeConfig(
            vectors={
                VECTOR_NAME: EdgeVectorParams(
                    size=EMBED_DIM,
                    distance=Distance.Cosine
                )
            }
        )
        _shard = EdgeShard.create(path=str(SHARD_DIR), config=config)

Putting it all together

So here is the full mental model, end-to-end:

• Every photo gets passed through CLIP's image encoder, producing a 512-dimensional embedding.

• Those embeddings get stored in Qdrant Edge, a vector database running entirely in-process on your machine

• When you search, your natural language query goes through CLIP's text encoder, producing a 512-dimensional query embedding

• Qdrant finds the stored image embeddings that are closest to that query embedding using cosine similarity

• Those closest matches are your results — ranked by semantic relevance, returned in milliseconds

OpenClaw — the brain

Having a search function is useful. Having an agent that can understand a natural language request, decide which tools to use, chain them together, and give you a coherent response — that is the difference between a script and something that actually feels like an assistant.

OpenClaw is a conversational agent framework that lets you define tools and gives a language model the ability to decide which tool to call, in what order, with what parameters, based on what the user says. It handles multi-turn conversation context, tool-calling logic, and response formatting, so you do not have to build any of that orchestration yourself. The tools we expose to the agent map directly to our REST endpoints:

This matters because real queries are rarely single-step. Consider: "find all my dog photos and check if any are duplicates." That is a search followed by a duplicate detection pass on the results. An agent handles that naturally, a plain search function simply cannot do it.

User: "Find all my dog photos and check for duplicates"

↓

Agent: calls /api/search {query: "dog"}

↓

Agent: calls /api/duplicates {threshold: 0.97}

↓

Agent: "🐕 Found 47 dog photos. Detected 3 duplicate clusters.

Deleting these 5 would free up space."

OpenClaw also means the interface is conversational from day one. You do not need to know the API. You just describe what you want, and the agent figures out how to get it.

Setting Up the Environment

All right, let us get our hands dirty. This section is about getting everything installed and verified before we write a single line of pipeline code. I want to make sure the foundation is solid, because nothing is more frustrating than debugging a setup issue halfway through a build.

Here is the full stack we are working with:

Step 1: Clone the repo and set up a virtual environment

Always work inside a virtual environment. We all know this, yet we also all have that one project from 2025 where we did not, and are still paying for it.

# Clone the repo
git clone https://github.com/vatsala-singh/AI-Powered-Photo-Search-and-Tagging-Agent.git
cd AI-Powered-Photo-Search-and-Tagging-Agent

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate        # Mac/Linux
# .venv\Scripts\activate         # Windows

Step 2: Install the dependencies

pip install -r requirements.txt

The requirements.txt covers everything: FastEmbed, Qdrant client, FastAPI, Uvicorn, Pillow, OpenClaw, and NumPy. The heaviest part of this install is FastEmbed pulling down the CLIP model weights the first time you run the app — about 150MB each for the vision and text towers. After that first download, everything is cached locally under qdrant-edge-data/models/ and never needs to re-download.

Also, it is worth knowing upfront: there is no GPU dependency anywhere in this stack. Everything runs on CPU. The FastEmbed library was specifically designed for this: it uses ONNX runtime under the hood, which is significantly faster than raw PyTorch inference on CPU.

Pro-Tip: If you are indexing an iPhone backup dump, you will likely encounter .heic formats. Standard Pillow doesn't support them natively out-of-the-box, which is why our script cleanly catches exceptions and logs them to avoid crashing hours of processing. If you want full HEIC support, ensure you add pillow-heif to your virtual environment dependencies.

Step 3: Understand the project structure

Before starting the server, it helps to know what you are looking at:

AI-Powered-Photo-Search-and-Tagging-Agent/
│
├── main.py                  # FastAPI app + OpenClaw agent entry point
├── config.py                # All configurable parameters in one place
├── requirements.txt
│
├── pipeline/
│   ├── embedder.py          # CLIP embedding logic (image + text)
│   └── indexer.py           # Batch photo processing and indexing
│
├── store/
│   └── qdrant_client.py     # Qdrant Edge setup and collection management
│
├── tools/
│   ├── search.py            # Semantic search tool
│   ├── tag.py               # Zero-shot auto-tagging tool
│   ├── duplicates.py        # Near-duplicate detection tool
│   └── albums.py            # Smart album grouping tool
│
├── test/
│   ├── embedder_test.py
│   ├── indexer_test.py
│   ├── search_test.py
│   └── qdrant_edge_client_test.py
│
└── qdrant-edge-data/        # Auto-created at runtime
    ├── storage/             # Qdrant's internal shard data
    ├── models/              # Cached CLIP model weights
    └── photos/              # Collection data

This is clean, flat, and easy to navigate. Each module has exactly one job, which makes the whole thing a lot easier to reason about when something eventually misbehaves.

Step 4: Glance at config.py

All the parameters that control system behaviour live in one place:

# config.py

CLIP_IMAGE_MODEL = "Qdrant/clip-ViT-B-32-vision"
CLIP_TEXT_MODEL  = "Qdrant/clip-ViT-B-32-text"
EMBEDDING_DIM    = 512          # CLIP ViT-B/32 output dimension

COLLECTION_NAME  = "photos"
QDRANT_PATH      = "./qdrant-edge-data"

BATCH_SIZE             = 32     # Images per indexing batch
TOP_K                  = 10     # Default search results returned
TAG_THRESHOLD          = 0.20   # Min similarity score for a tag to apply
DUPLICATE_THRESHOLD    = 0.97   # Min similarity to flag as duplicate

A few of these are worth understanding now rather than later:

TAG_THRESHOLD = 0.20 — This might look surprisingly low. Zero-shot classification works by computing similarity between an image embedding and a text label like "outdoor" or "food". Because these are short, generic labels rather than full descriptive sentences, similarity scores tend to sit in a lower range than a full natural language query would. 0.20 is calibrated to catch real matches without being too noisy. You can tune it up if you are getting too many irrelevant tags on your library.

DUPLICATE_THRESHOLD = 0.97 — This is intentionally very high. Two completely different photos of the same sunset can score 0.85 similarity. We only want to flag true near-duplicates — burst shots, accidental double-captures — so we set the bar close to 1.0.

BATCH_SIZE = 32 — Embeddings are generated in batches of 32 images at a time. This is the sweet spot for CPU throughput with FastEmbed. Smaller batches under-utilise the model; larger batches start to hit memory pressure on modest hardware.

Step 5: Start the server

uvicorn main:app --reload --port 8000

On first startup, FastEmbed will pull down the CLIP vision and text model weights if they are not already cached. You will see download progress in the terminal. Once that is done, which only happens once, startup is instant on every subsequent run. Once running, you have two ways to interact with the system: Option A: Interactive API docs (easiest for testing) Open your browser at http://localhost:8000/docs. FastAPI auto-generates a full Swagger UI, where you can fire requests directly without writing any curl command. Great for poking around during development. Option B: OpenClaw conversational agent This drops you into a chat interface where you talk to the agent in plain English. We will use this a lot more once the indexing and search pipeline are in place.

Step 6: Verify the setup

Before we build anything, let us confirm that the pieces are talking to each other:

# Quick sanity check - should return {"status": "ok"} curl http://localhost:8000/health

Also, to verify Qdrant Edge initialised correctly and the photos collection created:

curl http://localhost:8000/api/status

You should see the collection name, vector dimension (512), and a count of 0 indexed photos — which is correct, since we have not indexed anything yet. If both of those come back clean, your environment is fully set up. Models are cached, Qdrant is running in-process, the API is live. We are ready to actually build the pipeline.

Building the Image Embedding Pipeline

This is where the system starts to take shape. The embedding pipeline is the first thing that runs on your photo library; it is what converts a folder full of JPEGs into a searchable index of meaning. Get this right, and everything downstream just works.

The pipeline lives across two files: pipeline/embedder.py, which handles the actual CLIP inference and pipeline/indexer.py, which handles walking a folder, processing images in batches and handing vectors off to Qdrant.

The embedder

embedder.py is the cleanest file in the project, and intentionally so. Its only job is to take an image path or a text string and return a 512-dimensional NumPy vector. Everything else is someone else's problem.

# pipeline/embedder.py

from fastembed import ImageEmbeddingModel, TextEmbeddingModel
from PIL import Image
import numpy as np
from config import CLIP_IMAGE_MODEL, CLIP_TEXT_MODEL

# Load once, reuse for the lifetime of the process
# Models are large (~150MB each) — we never want to reload them per request
_image_model = ImageEmbeddingModel.from_pretrained(CLIP_IMAGE_MODEL)
_text_model  = TextEmbeddingModel.from_pretrained(CLIP_TEXT_MODEL)

def embed_image(image_path: str) -> np.ndarray:
    """Convert an image file to a 512-d CLIP embedding."""
    image = Image.open(image_path).convert("RGB")
    embeddings = list(_image_model.embed([image]))
    return np.array(embeddings[0])   # shape: (512,)

def embed_text(query: str) -> np.ndarray:
    """Convert a text string to a 512-d CLIP embedding."""
    embeddings = list(_text_model.embed([query]))
    return np.array(embeddings[0])   # shape: (512,)

A couple of things worth noting here. First, the models are initialised as module-level globals; _image_model and _text_model are loaded once when the module is first imported, and then shared across every subsequent call. This is a deliberate design choice. Each model is around 150MB and takes 2–5 seconds to load from disk. If you reloaded them on every embedding call, a batch of 500 photos would spend more time loading models than actually embedding anything. Second, the .convert("RGB") call on image load is not optional. Phone galleries are full of PNGs with alpha channels, HEIC exports, and the occasional oddly-encoded file. CLIP expects three-channel RGB input, and silently getting a four-channel RGBA tensor into the model produces garbage vectors. The conversion is two extra characters that save a lot of confusion.

Third, both vectors are L2-normalised by FastEmbed automatically. This is what makes the cosine similarity calculation downstream reduce to a simple dot product: fast, numerically stable, and consistent across all queries.

The indexer

indexer.py is where the real work happens. It takes a folder path, recursively finds all the images, processes them in batches of 32, and hands everything off to Qdrant.

pipeline/indexer.py

import os import uuid from pathlib import Path from datetime import datetime from PIL import Image

from pipeline.embedder import embed_image from store.qdrant_client import get_shard from tools.tag import generate_tags from config import BATCH_SIZE

SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".tiff"}

def index_folder(folder_path: str) -> dict: 

""" Recursively index all images in a folder into Qdrant Edge. Returns a summary: total found, indexed, skipped. """ 

folder = Path(folder_path) shard = get_shard()
image_paths = [
    p for p in folder.rglob("*")
    if p.suffix.lower() in SUPPORTED_FORMATS
]

total    = len(image_paths)
indexed  = 0
skipped  = 0
batch    = []

for i, path in enumerate(image_paths):
    try:
        vector   = embed_image(str(path))
        tags     = generate_tags(str(path))
        img      = Image.open(path)
        
        point = {
            "id":      str(uuid.uuid4()),
            "vector":  vector.tolist(),
            "payload": {
                "filename":  path.name,
                "filepath":  str(path.absolute()),
                "tags":      tags,
                "timestamp": int(path.stat().st_mtime),
                "width":     img.width,
                "height":    img.height,
            }
        }
        batch.append(point)
        indexed += 1

    except Exception as e:
        print(f"Skipping {path.name}: {e}")
        skipped += 1

    # Flush every BATCH_SIZE images
    if len(batch) >= BATCH_SIZE:
        shard.upsert(points=batch)
        batch = []
        print(f"  Progress: {i+1}/{total} images indexed...")

# Flush remaining
if batch:
    shard.upsert(points=batch)

return {"total": total, "indexed": indexed, "skipped": skipped}

The batch flush pattern — upserting every 32 images rather than one by one, or all at once at the end — is the right tradeoff for a few reasons. Upserting one image at a time means 500 database writes for a 500-image folder, which is slow. Accumulating everything in memory before a single final upsert means if the process dies at image 487, you lose all progress. Batches of 32 give you reasonable write efficiency with natural checkpoint behavior.

The try/except around each image is also not just defensive boilerplate. Phone galleries are genuinely messy: corrupted files, half-downloaded images, screenshots with unusual colour profiles, HEIC files that Pillow cannot parse without extra plugins. Logging and skipping is the right call rather than crashing the entire indexing run on one bad file.

Indexing Images into Qdrant Edge

We touched on Qdrant Edge in the tools section, and the indexer already calls into it. But I want to spend a proper moment here on how the database is actually set up, because the decisions made at this layer directly affect how well search and filtering work downstream. Getting the schema right once means you never have to re-index your entire library because you forgot to store something useful.

How Qdrant Edge works here

The standard Qdrant library you might have seen in other tutorials runs as a separate server process. You start it, it listens on a port, and your Python code talks to it over HTTP or gRPC. That is fine for production services, but it is overkill for a personal tool running on a laptop, and it violates my "no servers to manage" constraint.

Qdrant Edge is different. It runs entirely inside your Python process: no binary to start, no port to open, no network to call. You import it, point it at a folder on disk, and it just works. The specific class we use is EdgeShard, which is Qdrant Edge's in-process storage and retrieval unit. Think of it as a database and query engine rolled into one object, living right inside your application.

Setting up the shard

In our repo, the entire setup lives in store/qdrant_client.py, and its job is to manage one singleton edge shard, creating it fresh on first run or reopening it from disk on subsequent runs — finally exposing a get_shard() function that other modules can use to access it.

    """
    Return the singleton EdgeShard, creating it on first call.
 
    - If SHARD_DIR does not exist → create a brand-new shard.
    - If SHARD_DIR already contains data → reopen it (no config needed).
 
    EdgeShard runs entirely in-process. No binary, no port, no network.
    """
    global _shard
    if _shard is not None:
        return _shard
    
    SHARD_DIR.mkdir(parents=True, exist_ok=True)
    # Detect whether this is a fresh shard or an existing one.
    # EdgeShard.create() fails if data already exists on disk.
    shard_has_data = any(SHARD_DIR.iterdir())
    
    if shard_has_data:
        print(f"[qdrant_client] Reopening existing shard at '{SHARD_DIR}'")
        _shard = EdgeShard.load(path=SHARD_DIR)
    else:
        print(f"[qdrant_client] Creating new shard at '{SHARD_DIR}'")
        config = EdgeConfig(
            vectors={
                VECTOR_NAME: EdgeVectorParams(
                    size=EMBED_DIM,
                    distance=Distance.Cosine
                )
            }
        )
        _shard = EdgeShard.create(path=str(SHARD_DIR), config=config)
        print(f"[store] Shard ready — vector: '{VECTOR_NAME}', dim: {EMBED_DIM}")
    return _shard

The create vs load split

The EdgeShard.create() will throw an error if it already has a directory with shard data in it; hence we explicitly check whether SHARD_DIR has any content before deciding whether to create a new directory or load the existing one.

The singleton pattern

The _shard global and early return ensure that no matter how many modules call get_shard() — the search tool, the tag tool, the indexer —- all running together concurrently share one shard instance. Loading an EdgeShard from disk takes a non-trivial amount of time; you do not want to do it on every function call.

close_shard() is not optional. EdgeShard buffers writes in memory for performance. If your process exits without calling close_shard(), those buffered writes may not make it to disk. We hook this into FastAPI's shutdown lifecycle so it always gets called cleanly when the server stops.

@app.on_event("shutdown") def on_shutdown(): close_shard()

The payload schema

Every indexed photo gets stored in the shard as a vector plus a metadata payload. Rather than scattering raw dictionary construction across multiple files, we define the payload shape once as a proper dataclass in schema.py from dataclasses import dataclass, field from typing import List, Optional @dataclass

from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class PhotoPayload:
    filename:str
    path:str
    tags: list[str] = field(default_factory=list)
    timestamp: Optional[str] = None
    width: Optional[int] = None
    height: Optional[int] = None
    
    def to_dict(self) -> dict:
        return {
            "filename": self.filename,
            "path": self.path,
            "tags": self.tags,
            "timestamp": self.timestamp,
            "width": self.width,
            "height": self.height
        }

Writing points to the shard

With the shard initialised and the payload schema defined, writing a photo to Qdrant looks like this:

from qdrant_edge import PointStruct
from store.qdrant_client import get_shard
from schema import PhotoPayload

shard = get_shard()

payload = PhotoPayload(
    filename  = "beach_sunset.jpg",
    path      = "/Users/me/Pictures/2024/Goa/beach_sunset.jpg",
    tags      = ["sunset", "beach", "outdoor"],
    timestamp = "2024-05-01T18:42:00",
    width     = 4032,
    height    = 3024
)

point = PointStruct(
    id      = "3f7a2b1c-8e4d-4f9a-b2c1-7d8e9f0a1b2c",
    vector  = {"image": vector.tolist()},   # named vector matching VECTOR_NAME
    payload = payload.to_dict()
)

shard.upsert(points=[point])

One detail: the vector is stored as a named vector {"image": [...512 floats...]} matching the VECTOR_NAME key defined in EdgeConfig. This is Qdrant Edge's way of supporting multiple vector spaces per point (useful if you later want to add, say, a separate text description embedding alongside the image embedding). For now we only use one, but naming it correctly from the start means you do not have to re-index to add a second vector type later.

The indexer flushes in batches of 32, so in practice shard.upsert() receives a list of 32 PointStruct objects per call rather than one at a time, but the structure of each point is identical to the above.

In both standard Qdrant and its embedded, on-device counterpart Qdrant Edge, a Point is the fundamental unit of data storage. Think of it as the equivalent of a row in a SQL database or a document in MongoDB, but completely optimized for vector search. It consists of 3 components: a unique id, a vector, and metadata (payload).

Automatic Photo Tagging

One more feature that I included in our project is automatic photo tagging, because why not! Pure vector similarity is certainly powerful, but it works entirely on its own, while tags give us a second dimension to filter on, making the result sharper and more precise when you need them to be.

The only catch was: I didn't want to tag any of my pictures manually. So I made the system do it at the time of indexing!

The idea: zero-shot classification

Zero-shot classification sounds fancy but the intuition is simple. Remember how CLIP maps both images and text into the same 512-dimensional vector space? We can exploit that to ask: "how similar is this image to the concept of 'beach'?" by computing the cosine similarity between the image embedding and the embedding of the word "beach". '

If the similarity crosses a threshold, the tag applies. No training, no labelled data, no fine-tuning. The model already understands these concepts from its pre-training on hundreds of millions of image-text pairs. We run this for a vocabulary of 50+ semantic labels at index time, and store whichever tags cross the threshold directly into the photo's Qdrant payload.

TAG_VOCABULARY = [ "sunset", "sunrise", "beach", "ocean", "mountain", "forest", "city", "night", "snow", "rain", "fog", "sunny", "cloudy", "dog", "cat", "bird", "people", "crowd", "portrait", "selfie", "food", "coffee", "restaurant", "travel", "architecture", "car", "road", "nature", "flowers", "trees", "indoor", "outdoor", "party", "celebration", "sport", "screenshot", "document", "text", "map", ]

The is not a fixed list; it lives in tools/tag.py, flexible enough to add any label as you choose, without the need for reindexing everything. You can also just re-tag the existing indexed photos without actually touching their vectors. Generating tags for a photo Here is the full tag generation function:

# tools/tag.py

from pipeline.embedder import embed_image, embed_text
from config import TAG_THRESHOLD
import numpy as np

TAG_LABELS = [...]  # full list as above

# Pre-compute label embeddings once at module load —
# no point re-embedding the same 50 words on every photo
_label_vectors = {
    label: embed_text(label)
    for label in TAG_LABELS
}

def generate_tags(image_path: str) -> list[str]:
    """
    Run zero-shot classification on an image.
    Returns a list of tags whose similarity to the image
    exceeds TAG_THRESHOLD (default: 0.20).
    """
    image_vector = embed_image(image_path)

    tags = []
    for label, label_vector in _label_vectors.items():
        similarity = np.dot(image_vector, label_vector)  # cosine sim on normalised vectors
        if similarity >= TAG_THRESHOLD:
            tags.append(label)

    return tags

Pre-computing label embeddings are important. The 50+ tag labels do not change between photos. If we called embed_text(label) inside the loop for every image, we would be re-embedding the same words thousands of times during a library indexing run. Instead, we compute all label vectors once when the module loads, store them in _label_vectors, and just run dot products from there. It is a small optimization that makes a meaningful difference at scale.

The threshold of 0.20 is intentionally low. It surprised me too when I first saw it. But zero-shot classification on short single-word labels produces lower similarity scores than full natural language queries. There is simply less semantic content in the word "beach" than in the phrase "sunset on a beach with gentle waves". Setting the threshold too high means legitimate tags get missed. At 0.20, you might occasionally get a borderline tag on an ambiguous image, but you rarely miss a genuine one. Tune it upward if your library is getting noisy; tune it down if tags feel sparse.

Tags at index time vs query time

Tags are generated during indexing and stored in the payload — not computed at search time. This is an important distinction. By the time you fire a query, every photo already has its tags baked in. Search is fast because all the expensive work happened once, upfront. Here is where generate_tags plugs into the indexer we built in Section 5:

def generate_tags_from_vector(img_vec: np.ndarray, threshold: float = 0.20, max_tags: int = 6) -> list[str]:
    """
    Generate tags for an image vector using zero-shot CLIP classification.
    Tags with cosine similarity above threshold are included (up to max_tags).
    
    This is a utility function used for generating tags during indexing
    or when you already have an image vector.
    """
    tag_vecs = _get_tag_vectors()
    
    scores = {
        tag: float(np.dot(img_vec, vec))   # both normalized → cosine similarity
        for tag, vec in tag_vecs.items()
    }
    
    tags = sorted(
        [t for t, s in scores.items() if s >= threshold],
        key=lambda t: scores[t],
        reverse=True,
    )[:max_tags]
    
    return tags

Using tags as filters

Tags become genuinely useful when combined with vector search. Qdrant Edge lets us filter by payload fields alongside similarity search, so instead of “find the 10 most semantically similar images to ‘sunset’”, we can ask “find the 10 most semantically similar images to ‘sunset’ that are also tagged outdoor and not tagged blurry.”

We implement this with an over-fetch strategy: retrieve 5× more results than needed, then filter in Python by the requested tags, and return the top k that pass. This keeps the query logic simple while ensuring you always get back the number of results you asked for:

def search_photos(query: str, top_k: int = TOP_K, tags: list[str] = None) -> list[dict]:
    #search photo library with a Natural language query
    #takes in query, no of results to be displayed, and a list of tags
    # returns list of dicts with photo metadata and relevance score
    print(f"[search] Received query='{query}' with tags={tags} and top_k={top_k}")
    shard = get_shard()
    query_vector = embed_text(query)
    
    # Over-fetch when tag filtering is requested to have enough candidates
    # after post-filtering by tags
    over_fetch_multiplier = 5 if tags else 1
    fetch_limit = top_k * over_fetch_multiplier
    
    results = shard.query(
        QueryRequest(
            query=Query.Nearest(query_vector.tolist(), using=VECTOR_NAME),
            limit=fetch_limit,
            with_vector=False,
            with_payload=True,
        )
    )
    print(f"[search] Found {len(results)} initial hits for query='{query}' with tags={tags}")
    
    hits = []
    untagged_hits = []  # Fallback results for images without tags
    
    for point in results:
        payload = point.payload or {}
        point_tags = payload.get("tags", [])
        
        result_dict = {
            "path": payload.get("path"),
            "filename": payload.get("filename"),
            "tags": point_tags,
            "timestamp": payload.get("timestamp"),
            "score": round(point.score, 4)
        }
        
        # Post-filter by tags if specified
        # (EdgeShard doesn't support complex filters, so we filter in Python
        # after over-fetching more results than needed)
        if tags:
            if point_tags and any(t in point_tags for t in tags):
                # Has tags and matches the filter
                hits.append(result_dict)
            elif not point_tags:
                # No tags yet (images not auto-tagged), save as fallback
                untagged_hits.append(result_dict)
        else:
            # No tag filter specified, include all results
            hits.append(result_dict)
        
        # Stop if we have enough tagged results
        if len(hits) >= top_k:
            break
    
    # If we don't have enough tagged results, include untagged ones that match the query
    if tags and len(hits) < top_k:
        hits.extend(untagged_hits[:top_k - len(hits)])
    
    return hits[:top_k]

In practice, this feels natural. "Show me sunset photos that are outdoor and not blurry" is a completely reasonable thing to ask, and the system handles it without you needing to know anything about how the filtering works under the hood.

In Qdrant Edge (and the standard Qdrant engine), running a hybrid search that combines a natural language query and tag filtering relies on an architecture called Single-Stage Filtering. Unlike a two-stage approach (where you find matching tags first and then search vectors, or vice versa), Qdrant intersects the vector index and the metadata payload index simultaneously during graph traversal. This ensures high speed and prevents accuracy loss, even on resource-constrained edge devices.

Building the Search Agent

When you type “beach sunset with nobody around”, here is what happens:

1. Your query gets passed through CLIP’s text encoder → a 512-dimensional vector.

2. Qdrant computes the cosine similarity between that vector and every image vector in the shard.

3. The most similar ones come back, ranked by score.

4. Optional tag filters narrow the results further.

curl --location 'http://localhost:8000/search' \
--header 'Content-Type: application/json' \
--data '{"query": "eiffel tower from rooftop","tags":[], "top_k": 1}'

Response

{
    "query": "eiffel tower from rooftop",
    "results": [
        {
            "path": "/Users/vatsalasingh/Documents/Datasets/tag_phot/photo-1638051017225-0d9fcca18cf4.jpg",
            "filename": "photo-1638051017225-0d9fcca18cf4.jpg",
            "tags": [
                "cloudy",
                "city",
                "rain",
                "screenshot",
                "architecture",
                "travel"
            ],
            "timestamp": "2021-12-09T17:27:56",
            "score": 0.2864
        }
    ]
}

Handling edge cases gracefully

A few things that come up in practice with real photo libraries:

Low-confidence results. If your library genuinely does not have what you are looking for, the top results will still be returned, just with low scores. It is worth surfacing the score to the user so they can judge relevance themselves rather than presenting every result as a confident match.

Very short queries. Single-word queries like “food” work, but they cast a wide net. The text encoder has less to work with and produces a more generic vector. Longer, more descriptive queries , “homemade pasta on a wooden table” , tend to produce significantly sharper results because the query vector is richer and more specific.

Queries that are actually tag filters. “Show me all my screenshots” is not really a semantic search; it is a tag lookup. You can detect this pattern in the agent layer and route directly to a tag filter query rather than a vector similarity search, which will be both faster and more precise.

Orchestrating it all with OpenClaw

We have a working embedding pipeline, a local vector database, semantic search, and automatic tagging. Each piece works independently and is accessible via a clean REST endpoint. But right now, using the system means knowing which endpoint to call, with which parameters, in which order.

That is fine for a developer poking around in Swagger UI. It is not fine for something you actually want to use every day.

This is where OpenClaw comes in. It takes everything we have built and wraps it in a conversational interface, so instead of constructing API calls, you just describe what you want, and the agent figures out the rest.

How OpenClaw works

OpenClaw is built around a simple but powerful idea: give a language model a description of available tools, and let it decide which ones to call based on what the user says.

You define your tools — name, description, parameters — and OpenClaw handles the intent recognition, tool dispatch, result interpretation, and response formatting. It also maintains conversation context across multiple turns, so follow-up questions like “now filter those by outdoor only” work naturally without you repeating the original query.

The configuration is markdown-based, which keeps things readable and easy to modify without touching any Python.

@app.post("/chat")
def chat(req: ChatRequest):
    """
    Conversational endpoint. Accepts user message and conversation history,
    returns agent's reply after processing with tools.
    """
    # Define tool functions that the agent can call
    def search_tool(query: str, top_k: int = 10, tag_filter: list = None):
        """Search photos by natural language query"""
        return search_photos(query=query, top_k=top_k, tags=tag_filter)
    
    def duplicates_tool(threshold: float = 0.97):
        """Find duplicate or near-duplicate photos"""
        return find_duplicates(threshold=threshold)
    
    def tag_tool(image_path: str):
        """Generate and update tags for a specific photo"""
        return generate_tags_from_vector(image_path=image_path)
    
    # Create the agent with tools
    agent = Agent(
        tools=[search_tool, duplicates_tool, tag_tool],
        system_prompt="""
        You are a personal photo assistant. You help users search, organize,
        and understand their local photo library. You have access to tools
        for semantic search, duplicate detection, and tagging.
        
        When helping users:
        - Use the search tool to find photos by describing their content
        - Use duplicates tool to find and clean up duplicate shots
        - Use tag tool to inspect or update tags for specific photos
        
        Always be concise and helpful. When returning photo results, 
        format them clearly with filenames, similarity scores, and tags. 
        Use emojis sparingly but helpfully.
        """
    )
    
    # Run the agent conversation
    response = agent.chat(
        message=req.message,
        history=req.history
    )
    return {"response": response}

Real interaction flows

Let me show you what actual conversations with the agent look like, because this is where the whole system clicks into place.

Why the agent layer matters

I want to step back here for a second because I think this point is easy to miss when you are deep in the implementation details.

The tools we built — search, tag, duplicates — are individually useful. But they are most useful when they work together, and deciding how to combine them for a given request is genuinely non-trivial. Should a query like “clean up my camera roll from last month” trigger a duplicate search, a time-filtered semantic search, or both? Should it ask for confirmation before suggesting deletions?

These are judgment calls. And the agent layer is where those judgment calls live, cleanly separated from the tool implementations themselves. If you want to change how the agent reasons about a request, you update the system prompt or tool descriptions. You do not touch the search logic or the Qdrant client.

That separation is what makes the system extensible. Adding a new capability — OCR for screenshots, face clustering or video frame indexing — means writing a new tool and registering it with the agent. The orchestration logic adapts automatically.

Running Everything Locally — And Why That Matters

At this point, the system is fully built. Let us zoom out for a moment before the final sections, because I think the local-first nature of this project deserves more discussions.

Nothing leaves your device. Full stop.

Every component in this stack — the CLIP models, the Qdrant Edge shard, the FastAPI server, the OpenClaw agent — runs entirely on your machine. When you index a photo, the pixels never touch a network socket. When you search, the query is processed locally. When the agent responds, it is working entirely with data that lives on your disk.

This matters more than it might seem for a personal photo library. Your gallery is not just holiday snapshots; it is screenshots of private conversations, medical documents you photographed for reference, financial receipts, moments with family. The idea of that data flowing through a third-party cloud indexing service, being stored on someone else’s infrastructure, powering someone else’s model training pipeline — that should feel uncomfortable. And yet that is exactly what every major cloud photo service does, somewhere in the fine print.

This system does not. Your data is yours.

What it actually takes to run this

I want to give you honest numbers rather than optimistic benchmarks, because real hardware varies.

Initial indexing (one-time):

Run the initial indexing overnight if your library is large. You only ever need to do it once.

Search latency (after indexing):

Under 100ms end-to-end. That feels instant.

Memory footprint at runtime:

Comfortable on any machine with 8GB of RAM. Workable on 4GB with nothing else running.

Disk storage:

Each indexed photo adds roughly 2–3KB to the Qdrant shard (512 floats × 4 bytes + payload overhead). A 10,000-photo library takes around 30–50MB of vector storage. Your original photos are untouched; the shard is purely the index, not a copy of your images.

What’s Next — Extending the System

The system we have built is already genuinely useful: semantic search, automatic tagging, duplicate detection, smart albums, all running locally with zero cloud dependency. But the same architecture naturally extends in several interesting directions without starting over or introducing any cloud dependency. Same embedding pipeline, same Qdrant shard, same agent layer — just new tools hanging off the same foundation.

Here are the extensions I am actively thinking about.

Face clustering

Instead of searching by scene or object, cluster photos by the people in them, without ever labelling a single face manually. Run a face detection model over your library, extract face embeddings, cluster them by similarity, and you end up with groups that likely correspond to recurring people in your life. One labelling session to name the clusters, and from then on the agent knows who is who. No cloud face recognition API, no biometric data leaving your device.

OCR for screenshots and documents

A huge fraction of most phone galleries is screenshots, recipes, addresses, flight details, conversations, receipts. This content is invisible to a vision embedding model because CLIP understands visual semantics, not text content. Running OCR at index time and storing extracted text in the payload changes that completely. Suddenly screenshots become searchable by their actual content, and you get a true hybrid retrieval system: vector similarity for photos, text match for documents, both in the same result set.

Video frame indexing

Sample frames from video clips at a fixed interval, embed each frame as an image, and index them into Qdrant alongside your photos with the parent video filename — and seek timestamp as payload. At search time, video frames will surface in the same result set as photos. No transcription needed. Just frames living in the same embedding space as everything else, making your entire camera roll — not just the stills — searchable.

Hybrid search: vector + keyword + metadata

Right now search is vector-based with tag filtering on top. The natural evolution is a proper three-way hybrid: semantic similarity for meaning, keyword/OCR match for document content, and structured metadata filters for constraints like date range, orientation, or location. Each filter narrows the candidate set before ranking. The result is retrieval that is precise in a way pure vector search cannot achieve alone.

Time and location aware retrieval

Most phone cameras embed GPS coordinates and precise timestamps into every photo via EXIF metadata. Extracting and storing this at index time gives you a rich structured filter layer for free, powering queries like “photos from within 5km of MG Road” or “everything from the week after my birthday” without any semantic search at all. Combined with vector similarity, it brings you closer to something that feels like actual memory retrieval rather than just photo search.

None of these require a different architecture. Each is a new tool added to the same agent, indexing new data into the same Qdrant shard, using the same CLIP embedding space as the foundation. That is the part I find most satisfying — not that everything is built, but that adding the next thing feels natural rather than painful.

Final Thoughts

I started this project with a simple frustration: I could not find a photo I knew I had taken. I ended it with something I use every single day.

But more than the tool itself, what I want to leave you with is the idea behind it. We are surrounded by unstructured personal data, photos, screenshots, voice memos, documents; and the tools available to make sense of it are either locked inside proprietary ecosystems or so primitive they might as well not exist. The gap between what is possible with today’s open-source AI tooling and what most people actually have access to is enormous.

This project is one small attempt at closing that gap.

If you build something with this, or extend it in a direction I have not thought of, I genuinely want to hear about it. That is the whole point of writing this out rather than just keeping it to myself. The most interesting versions of this system are probably ones I have not imagined yet, built by people with different libraries, different problems, different constraints.

Drop a comment, reach out, or just build it and see where it goes.

Until next time.

First published on medium.

References & Further Reading

• Qdrant Edge Documentation

• Qdrant Documentation

• OpenClaw GitHub

• OpenAI CLIP Research Paper

• FastEmbed — Qdrant’s Lightweight Embedding Library

• Hugging Face Transformers