Clockwork Labs
116 posts
@clockwork_labs
Creating beautiful machines. Currently working on https://t.co/rUQoxP1R8d and https://t.co/ZIpXxgJDMZ
This is a dangerous and fundamental misunderstanding of how SpacetimeDB works. By default, SpacetimeDB waits for all data to be durably persisted to disk BEFORE exposing it to clients. The window in which you can lose data not "very small", it's absolutely 0. SpacetimeDB can be configured to expose data to clients before data is persisted, but so can Postgres (synchronous_commit = off). The genius of SpacetimeDB is that although we fsync asynchronously, we buffer all client reads (and write acks) until we've guaranteed they are persisted to disk. Not only does this guarantee durability of transactions from the perspective of clients, it also allows us to decouple fsync latency and fsync throughput. Pipelining is a powerful optimization. @mihir20/understanding-synchronous-commit-in-postgresql-54cb5609a221" target="_blank" rel="nofollow noopener">medium.com/@mihir20/under…
That being said, it's understandable how people who spent their careers just operating Postgres instead of designing their own database engine could be so confused."The WAL is asynchronous, and is flushed periodically to disk on the background (by default, every 50ms)." these kinds optimizations are appealing because the "window" at which you could lose data is very small. usually, though, production workloads are continuously writing data. when continuously writing, you're actually *guaranteed* to lose *some* data each time your host loses power. this is generally not desirable.
