Badrish Chandramouli

We are looking for senior Ph.D. students for a summer internship in the storage, indexing, concurrency, transactions space (data systems group, MSR Redmond, in-person). Ping me if you are interested, and apply at apply.careers.microsoft.com/careers/job/19…. Thanks!

@nonlinear_james @_neonsunset Let us know what is missing for your use case and it might be possible to add ... cc @kevinmontrose

@badrishc @_neonsunset It has *some* lua support. It, for instance, is not far enough along to work with the DistributedLock library.

Came across a research covering Garnet (C# reimplementation of Redis). Would you look at these numbers 👀 diva-portal.org/smash/record.j…

@nonlinear_james @_neonsunset Garnet has had Lua support for quite a while now.

@_neonsunset It needs lua because there are a ton of things that require it so can’t replace redis with it yet.

@AGuyReborn @andy_pavlo @databricks @SnowflakeDB @duckdb Fantastic summary! Loved reading it.

@andy_pavlo @databricks @SnowflakeDB @duckdb A fantastic read, happy new year!

Buckle up because we're crashing into the new year with my annual database retrospective: License change blowbacks! @databricks vs. @SnowflakeDB gangwar! @DuckDB shotgun weddings! Buying a college quarterback with database money for your new lover! cs.cmu.edu/~pavlo/blog/20…

garnet-server, Microsoft's RESP (Redis-alike) server built on the "FASTER" tech is now available as a .NET global tool, making it one line to install it, then just "garnet-server" to run it. Very nice for RESP geeks: nuget.org/packages/garne…

The Data Systems group at MSR Redmond @MSFTResearch is now looking for in-person summer 2025 interns in the data systems and exploration areas! Interested PhD students (US and international) can apply at jobs.careers.microsoft.com/global/en/job/… -- thank you!

@rluvaton @dexcompiler It's not open source at this point, we are yet to finalize the next steps for the project.

@dexcompiler @badrishc I really hope so

At #VLDB2024, check our Bf-Tree, our high-perf B-Tree design optimized for small key-values. It uses a mini-page abstraction to cache reads/writes and a variable-length buffer pool to maintain them. See vldb.org/pvldb/vol17/p3… and attend session C3 at 3:30pm today to learn more!

@MarkCallaghanDB @sunbains Thanks for the pointer Mark, I was not aware of the InnoDB work. Will look into it.

@FilasienoF @MarkCallaghanDB @sunbains No, this work is newer and is still internal for now until we figure out the next steps.

@FilasienoF @MarkCallaghanDB @sunbains Transactionally moving data between disk/memory is quite fascinating indeed! Also check our recent related work at arxiv.org/abs/2305.01516. The C++ code of F2 is open source (still a PR) in the FASTER repo: github.com/microsoft/FAST… (this is a hash based KV though, not a B-Tree).

Always surprised by Microsoft Research; this is very interesting

@FilasienoF @sunbains Thanks! Your comment re: relevance to paged structures is spot on. The Bf-Tree was inspired by our prior work in the other direction: FASTER's hybrid log optimizes record-granularity paging for external hash KV. We wanted to get similar benefits for page-oriented B-Tree designs.

As you underlined it is a interesting set of changes “inspired” by many other papers. Beating a heavily optimised BTree at point queries AND an optimised LSMTree in write throughput is not easy. As you mentioned the key change is in the buffer manager/leaf page, but the data structure doesn’t change a lot as you mentioned; this is one of the merits of the paper. I find interesting that you could apply the same set of techniques to other traditional paged data structures (what if we apply the same techniques to an external hash table?); it makes you think many “what if … ?”. But again very bald claims.

@jodydonetti @marcgravell @davidfowl Related to cold start -- FASTER and Garnet support checkpoint/recovery so you could have your cache instantly rehydrate after a process failure. Not sure IDistributedCache supports this, but we have seen this to be useful in other scenarios.

@marcgravell @davidfowl That would be oh-so-interesting to explore, and it would completely annihilate the cold start problem in one sweet move! But I’m wondering about paying the (de)serialization price for every operation. What I’m thinking about from some time instead is…

The changes to redis licensing might make github.com/microsoft/garn… an interesting alternative.

@marcgravell @jodydonetti @davidfowl You can use FASTER to store arbitrary live C# objects directly as well, using the GenericAllocator. That will allow reading and mutating without cost. You will however pay the serialization cost when spilling to disk, or reading from disk.

@jodydonetti @davidfowl Deserialize per operation: yeah, that's a complex one because I'm also torn on the topic of "not obviously immutable types" / "defensive copies"; current plan is to observe [ImmutableObject] to avoid per-fetch deserialize, but indeed: can't use FASTER to store live objects.

@davidfowl Damn, Garnet sitting on top of performance benchmarks when compared to Redis, KeyDb and Dragonfly ... If this doesn't scream Microsoft is dead serious about performance of .Net Core, I don't know what does. microsoft.com/en-us/research…

@neuecc @jodydonetti @marcgravell Yes, the performance with Garnet custom commands would be extremely high for this use case. Second best would be multi/exec (which by design incurs a lot of repeated parsing and bookkeeping in the critical path). And don't get me started on Lua ... :)

@jodydonetti @marcgravell Instead of LUA, would custom commands work? (This article is in Japanese, but for reference: neue.cc/2024/03/19_Gar… ). It would be limited to self-hosting, but the performance seems like it would be better than LUA.

Let's see if we can make the new Garnet usable as an IDistributedCache implementation. This would skyrocket its usage and allow it to be used from dependent projects, like FusionCache 😬 cc @marcgravell github.com/microsoft/garn…

@marcgravell @jsneedles @jodydonetti @neuecc The fact that you used the serialization API means there is still a lot of gas left in the tank for improvement. For blittable keys and values, we have versions that do not require serializers at all. And those code paths are way faster, particularly for disk. This is exciting!

@jsneedles @jodydonetti @neuecc todo: I want to overhaul the serialization code; also, I need to feed back to the FASTER folks that their serialization API is "old school" - could do with IBufferWriter<byte>/ReadOnlySequence<byte> support