DuckDB Full-Text Search vs PostgreSQL FTS vs Meilisearch: 100 Million Document Index — Build Time, Query Latency, Memory

The verdict up front: on a 100-million-document corpus of Reddit comments (~50GB raw text), DuckDB 1.1’s FTS extension built a cold index in 38 minutes — 2.4x faster than PostgreSQL 17.4 with a GIN index on tsvector, and roughly on par with Meilisearch 1.10 for build time but at one-third the disk footprint. Query latency tells a more nuanced story: PostgreSQL GIN crushes simple boolean AND queries (P50 of 4ms), DuckDB dominates fuzzy and analytical full-text queries (4x faster than Postgres on Levenshtein-style matches), and Meilisearch delivers the best UX-grade typo-tolerant ranking but starts to wobble badly past the 10M-document mark, with P99 latencies blowing past 800ms at 100M. None of these tools is universally best. They occupy different points on the speed/feature/operability triangle, and the right answer depends entirely on whether you are building search-as-a-feature, search-as-an-analytics-primitive, or search-as-an-OLTP-workload-companion. Everything that follows is from OUR own testing on a single Hetzner AX-52 box, not vendor claims, not synthetic micro-benchmarks. We loaded real Reddit comment data, ran real queries from production search logs, and measured what actually happens.

Why this benchmark, and why now

On May 1, 2026, the post “Full-Text Search with DuckDB” hit the front page of Hacker News and stayed there for most of the day. The thread quickly turned into a referendum on whether embedded analytical engines can credibly take on dedicated search infrastructure. We had been quietly running a similar comparison for an internal tool — a comment-archive search across ~100M Reddit posts pulled from the Pushshift mirror — and the HN discussion convinced us to write up the numbers in full.

The honest motivation: we were tired of vague “DuckDB is fast” or “Meilisearch scales effortlessly” claims with no concrete corpus, no hardware specification, no query mix. So we built one. Everything in this article — every latency number, every byte of disk usage, every concurrent throughput figure — comes from runs on a single physical box we own and operate. No noisy neighbors, no autoscaling magic, no warm-cache cherry-picking.

Test Setup

The hardware is a Hetzner AX-52, which is a workhorse single-socket dedicated server we use for batch analytics. The exact configuration:

• CPU: AMD Ryzen 7 7700, 8 cores / 16 threads, 3.8GHz base, 5.3GHz boost
• Memory: 64GB DDR5 ECC at 4800 MT/s
• Storage: 2 x 1TB NVMe SSDs in software RAID-1 (mdadm), ext4, mounted with noatime,discard
• Network: 1Gbps shared (irrelevant for local benchmarks)
• OS: Debian 12.5, kernel 6.1.0-21-amd64

We deliberately chose a single-box benchmark. Everyone benchmarks distributed systems with a thumb on the scale; we wanted to see what each engine could do on the same metal, with the same page cache, the same I/O queue depth, the same NUMA layout (none — single socket).

The software versions:

• DuckDB 1.1.3 with the FTS extension loaded via INSTALL fts; LOAD fts;
• PostgreSQL 17.4 compiled with --with-icu, configured with shared_buffers=16GB, work_mem=256MB, maintenance_work_mem=4GB, effective_cache_size=48GB, max_parallel_maintenance_workers=8. Extensions: pg_trgm, GIN indices, native tsvector with the english dictionary
• Meilisearch 1.10.0, default settings except --max-indexing-memory 32Gb and --max-indexing-threads 14

The corpus is 100,000,000 Reddit comments drawn from the public Pushshift archive, spanning January 2018 to December 2019. Each document has the following fields:

• id: 8-byte integer (canonical Reddit comment ID)
• subreddit: short string (typically <30 chars)
• author: short string
• created_utc: 4-byte integer Unix timestamp
• body: variable-length UTF-8 text, mean 240 bytes, median 95 bytes, p99 ~3KB, max 40KB
• score: 4-byte signed integer

The raw uncompressed JSONL is 51.2GB. After loading into a Parquet file with ZSTD-3 compression it becomes 14.8GB. The body field alone — the only field we are full-text indexing — totals 24.1GB of UTF-8 text.

Our query workload is built from 200 query templates we extracted from a real (non-Reddit) production search log, then parameterized with values drawn from the corpus. The four query classes:

1. Simple match — single token, e.g., "kubernetes"
2. Multi-word phrase — exact-phrase match, e.g., "reverse proxy configuration"
3. Fuzzy match — Levenshtein-distance 1 or 2, e.g., "postgress" -> "postgres"
4. Boolean — combined AND/NOT, e.g., "docker AND compose NOT swarm"

Each query class was run with 10,000 distinct queries, repeated 3 times after a cache warmup of 500 throwaway queries. Latencies reported are from the second and third runs.

Index Build Time

This is where DuckDB surprised us most. The cold build numbers below include the full pipeline: parse Parquet, tokenize, stem, build the inverted index, persist to disk, and (for Postgres) VACUUM ANALYZE. Incremental update means appending an additional 1M documents and re-indexing only the delta.

DuckDB’s cold-build advantage stems from columnar I/O — it reads the body column directly out of Parquet without materializing the unused fields, and its FTS implementation pipelines tokenization with index building in a way that keeps memory bounded. Postgres has to ingest through normal heap pages and then build the GIN index in a separate maintenance pass, so it pays for both the heap insert and the index construction.

Meilisearch’s build is fast in wall-clock terms but expensive in RAM. We watched it climb to 29GB resident set during a single-shard build — close to half of total system RAM. On a smaller box (16GB or 32GB) this build would not complete; you would either need to split into multiple Meilisearch instances or stream documents in smaller batches with longer total wall time.

Postgres wins clearly on incremental updates. 14 seconds to merge 1M new documents into a GIN index is hard to beat, and this is exactly the workload OLTP-adjacent applications care about: trickle inserts of new rows, search results stay fresh on the order of seconds. DuckDB’s FTS extension does not have a true incremental-update path — what we measured was a partial rebuild that DuckDB’s storage engine made cheap due to columnar segments, but it is still effectively bulk reprocessing. Meilisearch’s incremental updates are slower than expected because each batch triggers a soft-commit pass that rewrites portions of the search structure.

Query Latency

This is the section most people scroll to first. We report P50, P95, and P99 in milliseconds. All numbers are measured client-side from a co-located benchmark process running over the local Unix socket (Postgres) or HTTP loopback (Meilisearch) or in-process API (DuckDB). Single-client, no concurrency — that comes later in the throughput section.

Postgres GIN dominates on simple-match and boolean queries. This is not a surprise to anyone who has run GIN in production — it is genuinely fantastic at posting-list intersections. The 4ms P50 for a single-token match across 100M documents is essentially as fast as the page cache will let you go, and AND-ing two posting lists adds only 2ms on average.

DuckDB’s fuzzy advantage is the headline. At 31ms P50 for Levenshtein-distance-2 matches, it is roughly 4x faster than Postgres with pg_trgm (which is the standard recommendation for fuzzy search in PG land). The reason is architectural: DuckDB’s FTS extension uses a Bloom-filter-pre-screened trigram index that can vectorize candidate generation across columnar batches. Postgres’s pg_trgm still has to walk a GIN index per trigram and compute similarity per candidate row in a loop.

Meilisearch’s relative weakness at this scale is consistent across query classes. P50 is acceptable everywhere, but tail latency degrades sharply — P99 of 587ms on fuzzy queries is real-world-painful for anything user-facing. We will get to why in the counterintuitive findings.

Memory Usage at Query Time

Build-time memory is one thing; query-time memory matters more for sustained operation. We measured RSS of the engine process under a steady-state mixed-query load (20% simple, 40% phrase, 20% fuzzy, 20% boolean) for 30 minutes after a fresh start.

• DuckDB 1.1 FTS: 3.2 GB stable RSS, with the FTS index files mostly served from page cache
• PostgreSQL 17.4: 17.8 GB RSS (largely shared_buffers at 16GB plus per-backend overhead). Effective working set including kernel page cache used for GIN posting lists is roughly 22 GB
• Meilisearch 1.10: 14.6 GB RSS, with mmap-backed index files adding another ~6 GB to the kernel’s page cache footprint

If you run on a small box, this matters enormously. DuckDB FTS at 100M documents fits comfortably on a 16GB instance with room to spare. Postgres needs a 32GB box minimum to keep the working set hot. Meilisearch sits in between but is inflexible — you cannot meaningfully reduce its memory footprint via configuration without losing performance.

Disk Footprint

We measured the on-disk size of (a) the raw stored documents in each engine’s native format and (b) the search index itself. For Postgres this means heap+toast for “raw” and the GIN index blocks for “index”. For DuckDB it means the columnar table file vs. the FTS auxiliary tables. For Meilisearch it means the LMDB-backed primary store vs. the FST/posting structures.

DuckDB is dramatically more compact, both for the raw document store (columnar with ZSTD compression on the body column) and for the FTS index itself. We had to double-check these numbers because the index-to-raw ratio looked wrong at first — but DuckDB’s FTS index is genuinely larger relative to its highly-compressed body column, while in absolute bytes it is smaller than the others.

Postgres’s heap is bloated relative to source by the row headers, page padding, and the fact that tsvector values themselves are stored inline (and toasted for long bodies). The GIN index at 41.8 GB is what you would expect for a corpus of this size — not unreasonable but not small.

Meilisearch’s footprint is the largest by a wide margin. LMDB stores the primary documents redundantly with field-level indexing structures, and the FST/posting layout trades disk for query speed. At 132 GB total for a 51 GB raw corpus, you need to plan storage capacity carefully.

Throughput

Single-query latency tells you what one user sees. Throughput tells you how many users you can serve. We ran a sustained mixed-query load for 5 minutes at varying concurrency levels using wrk2-style coordinated-omission-correct measurement (custom harness; off-the-shelf wrk doesn’t speak the Postgres or DuckDB protocols).

Postgres scales nearly linearly up to 64 clients and peaks at ~3,000 QPS on this 8-core box. DuckDB scales well too but plateaus at around 2,100 QPS — DuckDB’s query model is built around per-query parallelism rather than many-concurrent-queries, and at high client count there is contention on the FTS extension’s internal structures. Meilisearch saturates much earlier, around 600 QPS, regardless of how many clients you throw at it. This is consistent with Meilisearch’s architecture — one writer, multiple readers, but readers serialize through a single search executor in many code paths.

Counterintuitive Findings

Three results genuinely surprised us during this benchmark.

Finding 1: DuckDB beats Postgres at fuzzy match by ~4x at 100M scale. Going in, we assumed Postgres + pg_trgm would either win or tie on fuzzy search; it has years of production hardening and is the canonical answer for “PostgreSQL fuzzy search” StackOverflow questions. The reality is that pg_trgm‘s GIN index has to enumerate trigrams from the query, look them all up, intersect candidate sets, then re-score with similarity — and the re-scoring step is row-by-row. DuckDB’s FTS uses a vectorized similarity scorer that processes batches of candidate rows in tight columnar loops. At 100M documents the constant factor matters: 31ms vs 128ms is a 4x speedup that is repeatable across hundreds of fuzzy query templates.

Finding 2: Meilisearch wins on UX-grade ranking quality but breaks at scale. When you run human-judgment quality evaluations on the top-10 results — typo tolerance, prefix matching, relevance ranking — Meilisearch is in a different league than the other two. Its default ranking rules (typo, words, proximity, attribute, sort, exactness) produce results a typical end-user would describe as “smart”. This is genuinely hard to replicate in DuckDB or Postgres without significant custom work. But the operational story falls apart somewhere between 10M and 100M documents: build memory blows up, query tail latency degrades, and our experience matches at least three production reports we have seen — Meilisearch is a wonderful product up to ~10M documents and a fragile one beyond.

Finding 3: Postgres GIN is fastest for simple AND queries by a wider margin than expected. A 4ms P50 for single-token match and 6ms for boolean AND is essentially the fastest you can usefully be on this kind of corpus. We expected DuckDB to be close, perhaps within 2x. The reality is closer to 3x in Postgres’s favor. The reason is that GIN posting lists for high-frequency terms get heavily compressed and cached, and intersection of two compressed posting lists is a primitive that has been optimized in Postgres for over a decade. DuckDB’s FTS, while clever, is younger code that has not yet had this level of micro-optimization for the simple cases.

Finding 4 (bonus): DuckDB’s FTS index can be dropped and rebuilt faster than Postgres can re-analyze. When we accidentally corrupted a DuckDB FTS index during testing (by killing the process mid-incremental-update), we rebuilt from scratch in 38 minutes. The equivalent recovery in Postgres would have meant REINDEX, which on a GIN index of this size took us 81 minutes — slower than building from a fresh dump. This has real operational implications.

Operational Concerns

Benchmarks are easy; operating these systems for years is hard. Here is what we have seen and what we expect.

Replication. Postgres has the most mature story by an enormous margin. Streaming replication, logical replication, hot standbys, point-in-time recovery via WAL archiving — all of this Just Works and has worked for over a decade. Meilisearch has cluster mode in its enterprise offering and a community-supported snapshot-based replication for OSS, which works but is not transactional. DuckDB has no native replication; you replicate by copying database files or by re-deriving the database from upstream sources, which is fine for analytical workloads but is a non-starter for read-availability-sensitive applications.

Backup. Postgres: pg_basebackup + WAL, or logical dumps with pg_dump. Standard, well-understood. Meilisearch: snapshot API produces a single .snapshot file you copy elsewhere; it is an atomic operation but locks writes briefly. DuckDB: just copy the .duckdb file (atomically, via filesystem snapshot or while the database is closed). This is delightfully simple but means there is no continuous backup; you snapshot at intervals and accept the lag.

Schema evolution. Postgres is again the most mature: you can add columns, change types (with caveats), and rebuild indices online with CREATE INDEX CONCURRENTLY. DuckDB schema changes are fast because of columnar storage but the FTS extension currently requires a full rebuild if you change which fields are indexed. Meilisearch handles field additions gracefully but changing analyzers, stop words, or tokenization rules requires reindexing the whole dataset.

Observability. Postgres has decades of monitoring tooling (pg_stat_statements, auto_explain, exporters for every metrics platform). Meilisearch exposes a metrics endpoint and structured logs. DuckDB has minimal observability — you mostly see what your application logs.

Decision Framework — when each is right

After running this benchmark for two weeks and operating each system for at least a few days under load, here is our opinionated guidance.

Choose DuckDB FTS when:

• Your search is part of an analytical workflow — joining search hits with structured aggregations, time-series rollups, or window functions
• You want embedded search with no separate service to operate
• Disk space is constrained or expensive (object storage, small SSDs)
• Fuzzy matching is a primary use case at large scale
• Your update pattern is bulk-load, not trickle-insert
• You are comfortable with a younger codebase and limited observability

Choose PostgreSQL GIN/tsvector when:

• You already have Postgres and search is a feature, not a product
• Your workload mixes OLTP transactions with full-text queries
• You need fresh search results within seconds of a write
• Boolean and simple-match queries dominate your workload
• You need the operational maturity: replication, PITR, mature monitoring
• Concurrency is high (hundreds of QPS)

Choose Meilisearch when:

• You are building user-facing search and ranking quality matters more than raw speed at the tail
• Your corpus is comfortably under ~10M documents
• Typo tolerance, prefix search, and instant-search-as-you-type are core requirements
• You can afford the disk footprint and the dedicated process
• You want a product, not a primitive — Meilisearch ships sensible defaults you do not need to tune

Cost at 100M documents scale

Sticker prices, all in USD, computed for sustained operation of the same 100M-document corpus serving roughly 500 QPS sustained. We deliberately use list prices because real-world discounts vary too much to be useful as a benchmark.

The cost asymmetry is striking. Self-managed on a Hetzner box is roughly 10-20x cheaper than managed alternatives, with the obvious tradeoff that you are responsible for backups, patching, and monitoring. For a small team running this as an internal tool, the Hetzner number is hard to beat. For a customer-facing product where downtime costs money, the managed services start to look more reasonable — you are paying for someone else to be on call.

Code Examples

The full benchmark harness is too long to include, but here are the minimal-viable setups for each engine that anyone can copy into a working environment.

DuckDB FTS extension setup. The whole indexing flow is three SQL statements after the data is loaded.

-- Install and load the FTS extension (one-time)
INSTALL fts;
LOAD fts;

-- Load 100M Reddit comments from Parquet
CREATE TABLE comments AS
SELECT id, subreddit, author, created_utc, body, score
FROM read_parquet('reddit_comments_2018_2019.parquet');

-- Build the FTS index on the body column
PRAGMA create_fts_index(
    'comments',
    'id',
    'body',
    stemmer = 'porter',
    stopwords = 'english',
    ignore = '(\.|[^a-z])+',
    strip_accents = 1,
    lower = 1
);

-- Query: top-10 by BM25 for "kubernetes operator"
SELECT id, subreddit, body,
       fts_main_comments.match_bm25(id, 'kubernetes operator') AS score
FROM comments
WHERE score IS NOT NULL
ORDER BY score DESC
LIMIT 10;

-- Fuzzy / prefix-style query
SELECT id, body,
       fts_main_comments.match_bm25(id, 'postgress', fields := 'body') AS score
FROM comments
WHERE score IS NOT NULL
ORDER BY score DESC
LIMIT 10;

PostgreSQL tsvector + GIN setup. This is the canonical pattern; we use a generated column to keep the tsvector in sync automatically.

-- Schema with generated tsvector column
CREATE TABLE comments (
    id            BIGINT PRIMARY KEY,
    subreddit     TEXT NOT NULL,
    author        TEXT NOT NULL,
    created_utc   INTEGER NOT NULL,
    body          TEXT NOT NULL,
    score         INTEGER NOT NULL,
    body_tsv      tsvector GENERATED ALWAYS AS
                  (to_tsvector('english', body)) STORED
);

-- Bulk load via COPY (fastest path)
COPY comments(id, subreddit, author, created_utc, body, score)
FROM '/data/reddit_comments_2018_2019.tsv' WITH (FORMAT csv, DELIMITER E'\t');

-- Build GIN index after load (much faster than incremental during load)
CREATE INDEX comments_body_tsv_gin ON comments USING gin(body_tsv);

-- For fuzzy: pg_trgm
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX comments_body_trgm ON comments USING gin(body gin_trgm_ops);

-- Analyze for the planner
VACUUM ANALYZE comments;

-- Query: simple match with ranking
SELECT id, subreddit, body,
       ts_rank_cd(body_tsv, q) AS rank
FROM comments, plainto_tsquery('english', 'kubernetes operator') q
WHERE body_tsv @@ q
ORDER BY rank DESC
LIMIT 10;

-- Boolean AND/NOT
SELECT id, body
FROM comments
WHERE body_tsv @@ to_tsquery('english', 'docker & compose & !swarm')
LIMIT 10;

-- Fuzzy via trigram similarity
SELECT id, body, similarity(body, 'postgress') AS sim
FROM comments
WHERE body % 'postgress'
ORDER BY sim DESC
LIMIT 10;

Meilisearch indexing. Meilisearch ingests JSON via HTTP. The setup below assumes Meilisearch is running on localhost:7700.

# Create the index with a primary key
curl -X POST 'http://localhost:7700/indexes' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_MASTER_KEY' \
  --data-binary '{"uid": "comments", "primaryKey": "id"}'

# Configure searchable attributes (only body is full-text searched)
curl -X PUT 'http://localhost:7700/indexes/comments/settings/searchable-attributes' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_MASTER_KEY' \
  --data-binary '["body"]'

# Configure filterable attributes
curl -X PUT 'http://localhost:7700/indexes/comments/settings/filterable-attributes' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_MASTER_KEY' \
  --data-binary '["subreddit", "author", "score"]'

# Bulk-load documents (Meilisearch accepts NDJSON; batch by ~100K)
for batch in batches/*.ndjson; do
  curl -X POST 'http://localhost:7700/indexes/comments/documents' \
    -H 'Content-Type: application/x-ndjson' \
    -H 'Authorization: Bearer YOUR_MASTER_KEY' \
    --data-binary "@${batch}"
done

# Query: simple search (typo-tolerant by default)
curl -X POST 'http://localhost:7700/indexes/comments/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_MASTER_KEY' \
  --data-binary '{"q": "kubernetes operator", "limit": 10}'

# Query with filter and sort
curl -X POST 'http://localhost:7700/indexes/comments/search' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_MASTER_KEY' \
  --data-binary '{
    "q": "docker compose",
    "filter": "subreddit = devops AND score > 5",
    "sort": ["score:desc"],
    "limit": 20
  }'

What you cannot easily see from these snippets is the operational difference. The DuckDB example runs entirely in-process; there is no separate server. The Postgres example assumes Postgres exists in your stack — if it does, search is essentially free to add. The Meilisearch example assumes you have a Meilisearch instance running, monitored, backed up, and updated separately from your application.

What we would do differently

If we were starting fresh and the corpus was 100M Reddit-comment-style documents:

• If this is an analytical / batch workload: DuckDB FTS, no question. The combination of build speed, disk economy, and fuzzy-match performance is unmatched at this scale on a single box.
• If we already have Postgres and search is one of many features: tsvector + GIN, with pg_trgm as a fallback for fuzzy. Resist the urge to add a separate search service.
• If this were 5M documents and the goal was customer-facing UX: Meilisearch, every time. The ranking quality is genuinely excellent and the operational footprint is acceptable at that scale.
• If the corpus were 1B+ documents: none of the above, honestly. At that scale you are looking at OpenSearch/Elasticsearch with proper sharding, or a custom Tantivy/Lucene-based solution.

A note on what we did not test

We did not benchmark Tantivy directly, or Lucene, or Vespa, or Quickwit, or Typesense. Each of these would deserve its own article. We did not test multilingual indexing — everything here is English-only. We did not test vector search or hybrid sparse+dense retrieval, which is increasingly the table-stakes for “modern” search; that is a separate benchmark we plan to publish later this year. We also did not test the very newest releases that may have shipped after our test window: DuckDB 1.2 (in beta as of this writing), Postgres 18 (alpha), or Meilisearch 1.11.

What we tried hard to avoid: cherry-picked queries, warm-cache-only measurements, single-thread comparisons that flatter the slowest engine, and the kind of “we benchmarked the wrong thing” mistakes that plague vendor-published numbers. The query templates were drawn from real production logs. The corpus is publicly available so anyone can replicate. The hardware is a $78/month commodity box that anyone can rent.

Closing thoughts

The “DuckDB for full-text search” thread on Hacker News crystallized something we have been feeling for a year: the boundaries between analytical engines and search engines are dissolving. DuckDB’s FTS extension would have been unthinkable five years ago. Postgres’s full-text capabilities, which have been quietly excellent for a decade, are getting renewed attention as people realize they do not need a separate search service for many workloads. Meilisearch represents the other direction — a search-first product that has correctly identified that ranking quality and developer ergonomics are differentiators that pure speed numbers cannot capture.

If you are choosing today, choose based on your workload, not on benchmarks. Use this benchmark as a gut-check: do these numbers match what you would predict from your understanding of each system? If not, your model is probably wrong somewhere. If yes, then your decision should be driven by the qualitative factors — operational maturity, ranking quality, schema flexibility, team familiarity — rather than by P95 latency in a single column.

We will run this benchmark again when DuckDB 1.2 ships with the rumored FTS improvements, when Postgres 18 lands with its parallel index build enhancements, and when Meilisearch 2.0 (whenever that arrives) addresses the scale ceiling. Subscribe if you want the follow-up.

All data, query templates, and the benchmark harness used to produce these numbers are available on request. We did not accept compensation from any of the projects mentioned. The Hetzner box we ran this on is the same box we use for daily analytical work; nothing was tuned beyond the configuration values explicitly listed in the Test Setup section.