I have been working on a project for a creator marketplace connecting brands, shoppers, and creators. The kind of product where a shopper opens the feed, scrolls through posts and short videos, discovers products, follows creators, saves things, clicks out to stores, and hopefully does not get shown the same thing every three swipes.

That last part sounds small until the feed starts getting real.

At first, the feed can get away with simple pagination:

give me the newest 20 things
then give me the next 20 things after that

That works when the feed is basically chronological. But the moment ranking enters the room, everything gets more interesting. Now the feed is pulling from followed creators, discovery candidates, trending content, vector similarity, short videos, and product-linked posts. The app is no longer just walking down one clean database index. It is assembling a page from multiple candidate pools.

And when that happens, repeats start sneaking in.

The actual problem

I did not want to mark something as “seen” just because the API returned it.

That would be too aggressive. Maybe the app fetched a page, but the user closed the tab before scrolling. Maybe the virtualized list loaded items below the fold. Maybe the network request finished, but the person never actually looked at the card.

So the rule became:

A post is only seen after the client reports that it was actually visible.

On web, that means an IntersectionObserver. On mobile, that means list viewability callbacks. Once a post crosses the visibility threshold, the client buffers an impression and flushes those impressions every few seconds.

The rough shape is:

visible post
  -> client buffers impression
  -> API validates and rate-limits it
  -> Redis stream stores it
  -> worker drains stream
  -> Postgres stores durable impression
  -> Bloom filter gets the media ID

The important bit is the order. Postgres is the durable source of truth. The Bloom filter is the fast “please do not show this again right now” layer.

Why a Bloom filter?

I needed a quick answer to this question:

has this user probably seen this media ID?

Not perfectly. Probably.

A normal database query would be accurate, but doing that for every ranking request and every candidate pool gets annoying fast. Feed ranking wants to ask this question over and over:

of these 100 candidate IDs, which ones has the user not seen?

A Bloom filter is good at exactly that kind of thing. It is memory efficient, fast, and slightly dramatic because it can say:

• “No, this item is definitely not in the set.”
• “Yes, this item is probably in the set.”

That “probably” is the tradeoff. Bloom filters can have false positives. An unseen post might get filtered out by mistake. But they do not have false negatives once the item is added. If the filter says an item is not present, it really is not present.

For a feed, that tradeoff is pretty reasonable. Accidentally skipping one eligible post is less painful than repeatedly showing something the user already scrolled past.

The per-user filter

I used RedisBloom and gave each signed-in user their own seen filter:

seen:{userId}

Each filter is reserved with:

capacity: 100000 items
error rate: 0.01
ttl: 30 days

That means each user gets a rolling memory of recently seen media. Not a permanent record. Not a full analytics system. Just enough memory to keep the feed fresh.

When the worker flushes impressions, it groups rows by user and bulk-adds the media IDs:

user A -> media 1, media 2, media 3
user B -> media 7, media 9

Then RedisBloom gets a batched add:

BF.MADD seen:userA media1 media2 media3
BF.MADD seen:userB media7 media9

Anonymous users are skipped because there is no stable user-level key. Their session-level dedup still helps ingestion, but the long-lived seen filter is for signed-in users.

Reading from the filter

When the feed builds a page, it fetches more candidates than it needs. That part matters.

If the UI asks for 20 posts, the ranker might fetch 100 discovery candidates, then ask the Bloom filter:

BF.MEXISTS seen:userA candidate1 candidate2 candidate3 ...

Anything that comes back as probably seen gets removed before ranking and formatting the response.

So the feed does not just ask the database for 20 and hope. It overshoots, filters, reranks, and then returns the best surviving items.

The filter is used on discovery and trending pools, where repetition is most likely to feel weird. The core idea is:

candidates
  -> remove in-page duplicates
  -> remove probably-seen items with Bloom
  -> rerank for relevance and diversity
  -> return page

The nice part: it follows real attention

The thing I like most about this setup is that it tracks actual attention instead of API output.

If the feed returns 20 items and the user only looks at 5, only those 5 go into the seen filter. The other 15 can still appear later. That feels fair.

It also works nicely with virtualized lists. A virtualizer can render, unrender, prefetch, and shuffle DOM nodes around. I do not want the backend making assumptions about what a person saw just because a query returned data.

The client is the only place that knows what crossed the viewport.

The less nice part: eventual consistency

There is a delay.

The impression has to be observed, buffered, sent, accepted, written to the stream, drained by the worker, persisted, and then added to the Bloom filter. In practice, that delay is fine, but it means the system is not trying to be magical.

If you refresh instantly after seeing something, it might still have a chance to appear before the worker catches up.

That is okay. This system is not a legal contract. It is a freshness nudge.

Mistakes this avoids

The Bloom filter is not replacing analytics. The durable impression log still exists in Postgres. That means I can still count views, recompute embeddings, inspect history, and rebuild things later.

The Bloom filter also is not used as the source of truth. If Redis falls over or the filter expires, the app loses some freshness memory, not the actual impression data.

That separation makes the system less precious:

Postgres = truth
Redis Stream = buffer
Bloom filter = fast memory

I like systems where each piece has one job and no piece has to be a hero.

The tradeoffs I accepted

Bloom filters are not free magic. I had to accept a few things:

• A false positive can hide something the user has not seen.
• A failed Bloom write can let something repeat.
• The filter needs RedisBloom support, not just plain Redis.
• The filter can underfill a page if too many candidates are already seen.
• The feed still needs fallback pools when filtering removes too much.

These are workable tradeoffs. The important part is knowing they exist instead of pretending the word “probabilistic” is just decoration.

The part I would improve next

The next version should probably include a repair job.

Since Postgres stores the durable impressions, I can rebuild a user’s Bloom filter from recent impression rows if the Redis key is missing, expired early, or suspiciously empty. That would make the fast layer disposable without making it fragile.

I would also add better observability:

• how many candidates Bloom removes per request
• how often pages underfill after filtering
• how often Bloom writes fail
• whether RedisBloom is available at startup

Those numbers would make it easier to tune capacity, fetch depth, TTL, and fallback behavior.

The tiny lesson

The best use of a Bloom filter is not “look, a clever data structure.”

It is more like:

I have a hot path that needs a fast maybe, and exact truth already lives somewhere else.

That is the shape that made it useful here.

In this creator marketplace, the Bloom filter is not the main character. It is a small, fast memory that sits beside the feed and quietly says:

you probably already showed this one
try something else

And honestly, that is a pretty good job for a tiny probabilistic data structure.