Harald Ng

34 posts

Harald Ng

Harald Ng

@haraldng

Ph.D. student at @KTHUniversity working on distributed systems and databases, Creator & Maintainer of OmniPaxos

Katılım Haziran 2020
63 Takip Edilen228 Takipçiler
Sabitlenmiş Tweet
Harald Ng
Harald Ng@haraldng·
Say hello to OmniPaxos! 👋 A cutting-edge #Rust library making your applications highly available, resilient, and flexible. More robust than Raft and VR, it thrives even in partial network partitions. Stay tuned for more insights into its internals! 🚀 🌐omnipaxos.com
English
0
13
49
9.3K
Harald Ng retweetledi
Audrey Cheng
Audrey Cheng@audreyccheng·
What is AI not good at? Database transactions! In our latest blog post, we dive into how AI could not find better scheduling algorithm than our VLDB '24 paper (in collaboration with @pbailis, @siobhcroo, @istoica05, and many others).
AI-Driven Research for Systems@ai4research_ucb

🎯 AI discovers an algorithm that makes database transaction schedules 34% faster [ADRS Blog #4] We revisit a classic database problem: dealing with transactional contention. Starting with a state-of-the-art algorithm from our VLDB '24 paper, we use OpenEvolve to automatically discover a new offline scheduling algorithm that significantly reduces makespan and boosts throughput! ✍️ Read the blog: adrs-ucb.notion.site/txn-scheduling 📖 ADRS Blog Series: ucbskyadrs.github.io 📃 VLDB Paper: vldb.org/pvldb/vol17/p2… 📄 ADRS Paper: arxiv.org/abs/2510.06189 👩‍💻 Code: github.com/UCB-ADRS/ADRS

English
2
4
23
6K
Harald Ng
Harald Ng@haraldng·
@eatonphil This is an excellent classification of how consensus is used in practice. Ultimately, consensus boils down to providing a distributed log. CfM uses the log to build a consistent service holding metadata, while CfD uses the log directly as a changelog/WAL.
English
0
0
4
97
Phil Eaton
Phil Eaton@eatonphil·
Really appreciate this post by Adam Prout. A good companion to Alex Miller's recent consensus design space study. @adamprout/categorizing-how-distributed-databases-utilize-consensus-algorithms-492c8ff9e916" target="_blank" rel="nofollow noopener">medium.com/@adamprout/cat…
Phil Eaton tweet media
English
2
31
262
15.5K
Harald Ng
Harald Ng@haraldng·
🚀Check out the latest tutorial on the OmniPaxos playground and dashboard! It runs on docker with an interactive dashboard in the terminal where you can input different network failures and KV-operations to see how OmniPaxos behaves. More info + video 🧑‍💻: omnipaxos.com/docs/playgroun…
English
0
8
36
3.4K
Harald Ng
Harald Ng@haraldng·
@kryoxnist In the post, we only read locally from a server. To ensure a read returns the latest committed value (i.e., linearizable read) can be implemented in several ways, such as using leader leases or treating the read as a write and committing it in the OmniPaxos log.
English
1
0
0
23
kryoxnist
kryoxnist@kryoxnist·
@haraldng Very nice! I was struggling to understand how a node knows it's the leader, so for a Get command it knows it's showing the most up to date record. As far as I understood the replication is async, right? How could a node ensure it is replaying the latest possible state for a key?
English
1
0
1
36
Harald Ng
Harald Ng@haraldng·
New blog post! We show how to use OmniPaxos to turn a single-server #RocksDB service in #Rust into a distributed version. OmniPaxos hides all complexities related to consensus and provides consistent replication through a simple log API 🚀🧑‍💻 Post + video: omnipaxos.com/blog/building-…
Harald Ng tweet media
English
2
13
54
5.9K
Harald Ng
Harald Ng@haraldng·
Hyped to be at #SOSP2023. Drop by the poster session tonight for a chat on how we make consensus proactive and efficient with workload-driven techniques using MetaCoordination 🚀💫
Harald Ng tweet media
English
0
0
5
790
Harald Ng
Harald Ng@haraldng·
@AlekseyCharapko I agree! Great to see changes to MultiPaxos that overcome one of the scenarios. Further modifications can probably be made to handle all scenarios in the paper, including the chained scenario. But that should not be left to engineers, especially if protocols are underspecified!
English
0
0
4
100
Aleksey Charapko
Aleksey Charapko@AlekseyCharapko·
MultiPaxos protocol is largely underspecified, so many practical design decisions are left in the air for engineers to figure out. Implementing MultiPaxos is hard. That being said, a sensible implementation seems to handle partial network partitions well.
English
2
2
12
2K
Aleksey Charapko
Aleksey Charapko@AlekseyCharapko·
State machine replication reliability is a big deal! Recently, OmniPaxos claimed that the "classical" MultiPaxos loses liveness under some partial network partitions. We felt something was off there and tested it with our MultiPaxos implementation. More: zhiying12.github.io/posts/2023/06/…
English
5
21
94
16.3K
Harald Ng retweetledi
Paris Carbone
Paris Carbone@SenorCarbone·
A great ACM recognition both for the groundbreaking data streaming research and innovations behind @ApacheFlink but also for the magnitude of what this system became with nothing less than the help of all the amazing minds and hard work of the whole open source community!
Fabian Hueske@fhueske

Big news: @ApacheFlink receives this year's @sigmod Systems Award! The project was started at TU Berlin back in 2009 but it would be nowhere today (feature and adoption-wise) without its awesome community. Thanks to everyone who contributed to Apache Flink! 👏🙏

English
0
1
27
1.3K
Harald Ng
Harald Ng@haraldng·
@eatonphil Yes! The decoupling of leader election and replication is crucial but not unique, most paxos-based/inspired protocols do that. The second and equally crucial part in OmniPaxos' leader election is to omit all unnecessary candidate restrictions and focus on quorum-connectivity.
English
0
0
1
66
Harald Ng
Harald Ng@haraldng·
@jorandirkgreef But for sure, borrowing some ideas from OmniPaxos to VR would probably both solve most issues and optimize for less view changes!
English
0
0
1
63
Harald Ng
Harald Ng@haraldng·
@jorandirkgreef intuitively, I'm a bit worried if servers directly send DoViewChange as soon as they think the leader has failed. Consider a scenario A-B-C-D-E where regardless of who is the leader, there is at least one server disconnected from it which triggers higher views causing livelock.
English
1
0
1
69
Harald Ng
Harald Ng@haraldng·
🚀 Excited to share our first blog post! Discover how OmniPaxos offers unmatched resilience to network failures by eliminating unnecessary restrictions in leader election and focusing on what is essential: quorum-connectivity. Details💡: omnipaxos.com/blog/how-omnip…
English
1
23
68
22.6K
Harald Ng
Harald Ng@haraldng·
@jorandirkgreef Thanks for a great catch! We will try updating the table and annotate that particular scenario for VR.
English
1
0
1
118
Joran Dirk Greef
Joran Dirk Greef@jorandirkgreef·
@haraldng “In VR, a server can only vote for a candidate (by sending DoViewChange) if itself is QC.” Whereas Cowling's own '12 VSR paper discusses this more broadly under 8.2: “We could avoid the round of StartViewChange messages by having replicas write the new view number to disk”.
English
2
0
1
203
Harald Ng retweetledi
Paris Carbone
Paris Carbone@SenorCarbone·
This distills the essence of partial connectivity, a greatly overlooked issue in today's consensus systems originally observed by @heidiann360 @ittaia a few years back. OmniPaxos shows how to boost liveness by decoupling leader election from Sequence Paxos-based log replication.
Harald Ng@haraldng

🚀 Excited to share our first blog post! Discover how OmniPaxos offers unmatched resilience to network failures by eliminating unnecessary restrictions in leader election and focusing on what is essential: quorum-connectivity. Details💡: omnipaxos.com/blog/how-omnip…

English
1
9
25
4.7K
Jim Dowling
Jim Dowling@jim_dowling·
Network partitions has long been the Achilles heel of Paxos/Raft. OmniPaxos is the first Paxos algorithm that works under network partitions. It is a major breakthrough in consensus algorithms for real-world systems, where network partitions are a fact of life.
Harald Ng@haraldng

Say hello to OmniPaxos! 👋 A cutting-edge #Rust library making your applications highly available, resilient, and flexible. More robust than Raft and VR, it thrives even in partial network partitions. Stay tuned for more insights into its internals! 🚀 🌐omnipaxos.com

English
2
8
38
5.5K
Harald Ng retweetledi
Paris Carbone
Paris Carbone@SenorCarbone·
⚡The future of high availability is here! Our #Rust library offers profound resilience, flexibility, & more. ☁️ Try OmniPaxos to replicate your apps today: omnipaxos.com . More juicy details to be revealed in the following week! @EuroSys_conf
Paris Carbone tweet media
English
1
5
23
1.1K
Harald Ng
Harald Ng@haraldng·
Thrilled to have presented my paper on UniCache at #EDBT2023 in Greece! Discussed how RSMs such as Raft and Paxos can learn from the workload to reduce the network I/O and improve performance. Full paper: openproceedings.org/2023/conf/edbt…
Harald Ng tweet media
English
0
1
16
649

Keşfet

@pbailis @siobhcroo @istoica05 @eatonphil @kryoxnist @AlekseyCharapko @ApacheFlink @jorandirkgreef