Pelikan Cache

Twitter Cache team just open-sourced Pelikan, a cache framework. More details on project website: pelikan.io

I talked about S3-FIFO at #SOSP today. S3-FIFO is a FIFO-based eviction algorithm that is simple, scalable, and efficient. For example, it can reduce the miss ratio of your LRU caches by up to 72% and improve throughput by 6x at 16 threads. S3-FIFO is being tried out at a few companies trying out, e.g., Google, VMWare, and a few startups. Come to talk to us if you are interested. paper: dl.acm.org/doi/10.1145/36… video: youtube.com/watch?v=5T2dHj… slides: jasony.me/slides/sosp23-… discussion: news.ycombinator.com/item?id=379993…

If you are attending QCon SF, check out Juncheng (@1a1a11a)'s talk on Ubiquitous Caching for the "Building Modern Backend" track at 1:40pm, where he discusses hardware trend, emerging workloads, and running large in-process caching using Segcache. qconsf.com/presentation/o…

I am still amazed at how easy it was to run @pelikan_cache on Ampere Altra-based Tau T2A VMs on GoogleCloud. @Brianiac was able to make it work instantly and make it fly in the matter of hours! 🙇‍♂️

Yep, we agree. betterprogramming.pub/software-compo… Corollary: If you do not know what *exactly* Pelikan Cache is, don't worry. It's by design.

🎙️E49 of the Open Source Startup Podcast is LIVE🎙️ Ft. @dymxzvf, Cofounder of caching platform @MomentoHQ & @thinkingfish from the Twitter team for @pelikan_cache ✨Check it out✨ anchor.fm/ossstartuppodc…

We know our documentation has a long way to go, so we are spending September, heads-down, to write about the design of Pelikan. If you want to known about anything in particular, please tell us in the reply. In October, we'll return with a daily tweet series on Rust + Pelikan.

Almost no where else in the infrastructure software would the following statement be more true than it is for caching: "Service delayed is service denied." Make it fast. Make it predictably fast. Make it consistently fast. #CachingWisdomOfTheDay

Friends don't let friends enable dynamic scripting in cache. All it takes is one bug or a poorly implemented algorithm to completely kill server performance. Scripting is useful but should be treated as other server code- tested, benchmarked, deployed. #CachingWisdomOfTheDay

@ksshams People may not want to, but they do (due to lack of alternatives sometimes), then they have to dedup upon LRANGE type of queries. But this is why data structures with position-based indexing only is less user-friendly than value-based indexing (e.g. sorted map).

@pelikan_cache for example, do you really want to retry list push?

You're probably familiar with read vs. write. But also consider: - idempotent vs. non-idempotent write: the former can be safely retried, not the latter; - regular vs. privileged command: e.g. you probably don't want people to run FLUSH_ALL willy-nilly. #CachingWisdomOfTheDay

Accessing a large in memory dataset is unlikely to be CPU cache friendly, even if it has a fairly skewed popularity distribution. What could / should be kept in CPU caches? IO buffers. How? CPU pinning, flow steering, and keeping storage threads away. #CachingWisdomOfTheDay

@tivrfoa Yes, chunking works, this was why Twitter added hybridlist (a linked list of zip list) to its forked Redis in 2012, an idea that got picked up by the OSS Redis and gave us quick list.

@pelikan_cache ... but probably a possible way to accomplish that would be to split the object in chunks. 2/2

In cache with data structures, writes tend to be incremental. It’s important to keep memory op proportional to update instead of object size. A DIMM carries ~40GB/s. If updates against 1MB objects involve memcpy whole objects, throughput will be <40K RPS. #CachingWisdomOfTheDay

@ksshams Twitter’s Redis fork does that. But what would be better is to offer more data structure in Pelikan (we are working on it)

@pelikan_cache I wish I could add an item to a list in Redis and set the TTL in one call. Alas, that doesn’t seem to be possible

Always have a TTL for anything put in cache. To list a few reasons: - it provides a well defined bound on data inconsistency; - it gives cache backend a strong hint on how to retain useful data for the right duration; - it helps you stay GDPR-compliant. #CachingWisdomOfTheDay

@ksshams The lack of standardized, representative (of production) benchmark suites allows many to game the numbers. May we suggest playing some (over 50 distinct workloads, in fact) real-world Twitter traces for a change? github.com/twitter/cache-…

@pelikan_cache Precisely. All I’m saying is that sometimes write can be more expensive than appears in latency of the write as it *could* have longer term implications

Cache writes are not always more expensive than reads. Simple read/write can achieve similar RPS because RPC costs dominate. There are some “tipping points” wrt RPC- e.g. you can expect sudden dip on RPS at MTU, socket buffer, and/or request buffer sizes. #CachingWisdomOfTheDay

@ksshams To be fair, if rewriting is handled in place, or allows immediate reuse of memory (e.g. if value sizes don’t change much) then fragmentation is less of a problem. Worst case scenario is a completely different story, of course.

@pelikan_cache This really depends on the memory layout of the cache. If you keep rewriting the same keys repeatedly, you can create fragmentation and cause the cache to slow down over time. Luckily, Pelikan Seg-Cache does a really good job with pre-allocating memory & using it effectively

@tivrfoa And yes, SLO for Service Level Objective, typically defined as good throughput with a particularly p99/p999 constraint.

@tivrfoa Mostly yes. TLS is used by most inter-service communication at this point. This was a direct outcome of trying to be GDPR compliant.

Threading model is especially important for secure cache access. While modern hardware has made encryption much faster, TLS frames add memory & CPU overhead. Worse still, TLS handshake is 10x more expensive than TCP– connect is a serious threat to SLO. #CachingWisdomOfTheDay