Sabitlenmiş Tweet
Fast general purpose transactions with optimal global latency in Apache Cassandra (and making today’s LWTs faster) m.youtube.com/watch?v=YAE7E-…
English
Benedict Elliott Smith 🇺🇦
1K posts
Benedict Elliott Smith 🇺🇦
@_belliottsmith
Apache Cassandra @apple
Katılım Mart 2014
1.1K Takip Edilen205 Takipçiler
@AlexMillerDB @penberg I think the phrasing “log position 42 is X” is broadly equivalent to having a leader-based protocol, however it is implemented.
English
@AlexMillerDB @penberg Leader-based protocols probably have an easier time enforcing a nice property here once a command reaches the leader, but I imagine that quorum-like optimisations for eg performing reads from followers *probably* fall prey to these same issues.
English
With a single-writer and multiple readers, can you guarantee linearizability with eventual consistent system such as Apache Cassandra or Scylla using read-write quorum? betterprogramming.pub/cassandra-cons… suggests yes, but I am having hard time convincing myself.
English
@AlexMillerDB @penberg Skimmed paper. It doesn’t seem their definition is much better than the best I’ve come up with; it would be nice to have a better formalisation than “its outcome is decided soon, preferably before the process crashes”. I like the idea of “strict strict serializablility” though…
English
@AlexMillerDB @penberg It’s worth noting that we have partially addressed this in Accord - if a transaction is recovered and known not to have been agreed then we abort it. If the “following” operation does not witness it then it is aborted, for instance, however “following” is ambiguous…
English
@AlexMillerDB @penberg Indeed, leaderless consensus protocols are worse as a write can submarine forever, whereas here the next write will resolve the situation. Personally I think this is a flaw in the definition of linearizability, but I’m not sure the best way to formalise an improvement.
English
@AlexMillerDB @penberg I think timeout is essentially an operation that has no defined end, the server indicates the operation’s status is unknown. This is also true of most leaderless consensus protocols, which are considered linearizable.
English
@AlexMillerDB @nevgeniev @jorandirkgreef @DominikTornow @ifesdjeen By data race here, I mean a pure simple local race on some shared memory location two threads are accessing concurrently. Not a general race condition where things just happen in the wrong order, of which of course this technique has found many.
English
@AlexMillerDB @nevgeniev @jorandirkgreef @DominikTornow We haven’t employed this capability much yet outside of whole-system testing for distributed consensus, but this has found at least one data race that I recall. Probably others I forget, too. I think @ifesdjeen also used it to demonstrate a race we had found through other means.
English
Is Deterministic Simulation Testing “mainstream”?
(Would ❤️ it to be!)
A deep dive discussion with @DominikTornow recorded last week.
x.com/dominiktornow/…
Joran Dirk Greef@jorandirkgreef
To be clear, Deterministic Simulation Testing is more than simply fault injection or chaos engineering (common misconception). To do DST, the “system under test” itself must also be written 100% deterministically. Very few DBMS systems (I know of only 3!) actually do DST. And that’s why @AntithesisHQ wrote literally a deterministic computer (again, not simply fault injection) to run non-deterministic binaries, deterministically. Even with our own DST as “internal audit function”, we have been using Antithesis as our “external audit function” at @TigerBeetleDB since 2022 (customer no. 13). Antithesis’ technology is a sea change for anyone coming from chaos engineering or non-deterministic simulation testing—it’s that different.
English
@AlexMillerDB @nevgeniev @jorandirkgreef @DominikTornow This technique should catch simple increments that don’t use eg atomic increment.
English
@AlexMillerDB @nevgeniev @jorandirkgreef @DominikTornow We don’t do anything super clever but we are able to explore a subset of data races. We byte weave in “nemesis” points and randomly stop/start threads at these points. This won’t find most bugs that rely on incorrect usage of the Java memory model, eg missing volatile keyword
English
@eatonphil The paper doesn’t take a position on this; a coordinator simply does not need to be a replica, which is an important property for protocol analysis. But in practice in Cassandra coordinators are replicas. I believe DataStax intend to disaggregate coordinators from replicas.
English
One question I have about Accord is on the Coordinators. Are they separate from replicas?
Sometimes it is made to sound like that.
Does that mean you're running some number of replicas and some number of coordinators?
English
Good time as any to revisit Cassandra's new leaderless consensus protocol, Accord, aimed at replacing Paxos inside of Cassandra. Many interesting critiques in here.
Introduced last year; brought back to my attention courtesy of Sam Lightfoot.
github.com/eatonphil/acco…
English
@eatonphil EPaxos has particularly poor properties under contention, as the exact same fast-path quorum must witness every contending transaction in the exact same order for the fast-path to be taken.
English
@eatonphil This section actually suggests that existing leaderless protocols perform poorly under contention, not that leader-based protocols do.
English
@jorandirkgreef @eatonphil We plan to, but I would prefer more crc32/64 (ie covering small chunks) as they provide hard guarantees for bit flip pattern detection that cryptographic hashes do not (a single bit flip may cause undetected corruption). Koopman is a great resource on CRCs checksumcrc.blogspot.com/2015/06/is-hd1…
English
@_belliottsmith @eatonphil Great question. Do you know anyone using 128-bit CRCs [1]?
Whereas Blake3 has become so fast, with the benefit of also being cryptographic—which we leverage for TB’s hash chain verification of the WAL. CRCs wouldn’t work for that.
[1] cf. github.com/rurban/smhasher
English
Interesting discussion in Slack today about the effectiveness of Ethernet/TCP/IP checksums.
They exist, but the guarantees they give are not particularly great!
No alternative but to do cryptographic checksums at the application layer (above TCP/IP).
noahdavids.org/self_published…
English
@AlexMillerDB @maximecaron @penberg Cluster config can be implemented on top of Accord itself, of course, but for now we will be using Cassandra’s (which is, uh, implemented on top of Paxos for now 🙈)
English
@AlexMillerDB @maximecaron @penberg We’d absolutely be open to contributions providing baseline implementations of these for use cases that want to just plug it in. The library itself today only comes with non-persistent implementations of everything, or in the case of cluster config only toy ones for testing
English
Cuz it came up in Discord recently and it's hard to search while avoiding blockchain. How I think about consensus:
Leader-based:
* Paxos/Raft
* Viewstamped Replication
Leaderless (or decentralized or peer-to-peer):
* CRDT
* Operational Transform
* CHORD
* IPFS
Sound right?
English
How big of a breaktrough is the new consensus protocol that Cassandra is adopting — Accord? It seems like the best of both worlds in a way that sounds too good to be true. I haven't read the paper yet.
thenewstack.io/an-apache-cass…
English