Public Notes & Digital Garden
Atomic knowledge, real-world gotchas, Kafka cluster tips, and evolving research observations.
[search]Mature
Anatomy of a Segment Merge: Minimizing Lucene I/O Spikes under Sustained Ingestion
A benchmark measuring write amplifications across TieredMergePolicy vs LogByteSizeMergePolicy. Investigating page cache thrashing, flush stall anomalies, and zero-drop throttling strategies.
2026-07-14
[Lucene]Mature
Lucene Term Dictionary FST vs DocValues
DocValues use column-oriented on-disk arrays memory-mapped into OS page cache. Always enable doc_values for fields used in sorting, aggregations, and script scoring.
2026-05-18
[Crawlers]Draft
Crawler Politeness Token Bucket with Redis ZSet
Store next_allowed_fetch_epoch in a sorted set per domain. Workers query ZRANGEBYSCORE -inf now LIMIT 1. Constant O(log N) scheduling.
2026-04-12
[Search]Evergreen
Elasticsearch Refresh Interval Pitfall
The default index.refresh_interval: 1s forces Lucene to generate tiny segment files every second. For ingestion-heavy jobs, raise to 30s or -1 during initial loads.
2026-03-05