PeerDB (YC S23)

📣 🎉 Big announcement coming your way! ClickHouse acquires PeerDB for native Postgres CDC integration. blog.peerdb.io/clickhouse-acq… We are super excited to join forces with ClickHouse to enable thousands of customers to seamlessly replicate data from Postgres to ClickHouse for real-time analytics. This blog captures our thesis around the acquisition and what it means for customers. We want to thank PeerDB customers, investors, and the entire community for supporting us through this journey. Means a lot to us! ❤️

Mind-blowing growth in migrations from Postgres since the @PeerDBInc team joined @ClickHouseDB last summer! 🚀 Try it out our Postgres CDC connector for yourself here: clickhouse.com/cloud/clickpip…

The growth in customers using Postgres CDC with @ClickHouseDB has been incredible to witness. This is exactly why I was so excited about ClickHouse's acquisition of @PeerDBInc! ❤️ If you're a Postgres user looking for blazing-fast analytics, we're here to make your journey seamless with ClickHouse—a purpose-built analytical database. Here’s the link to get started with Postgres CDC to ClickHouse: clickhouse.com/cloud/clickpip… And let's not forget—we're just getting started! 🚀

🎉 Migrating or replicating your Postgres database to ClickHouse? This blog dives deep into best practices for modeling your data and queries to get the most out of ClickHouse. Read the blog: buff.ly/yFZK117 🔥 What you’ll learn: 🌲 ReplacingMergeTree –Learn how this table engine enables you to efficiently handle data mutations and deletions while still ensuring blazing-fast ingestion performance. 🔄 Deduplication Strategies – Learn how to effectively use the FINAL keyword to remove duplicates and discover ways to minimize migration effort using views and refreshable materialized views. 🔑 Custom Order Keys – Key considerations to keep in mind when choosing ordering keys for your tables. 🔗 JOINs – Tips to optimize JOINs in ClickHouse and efficiently handle the Postgres 3NF model. 🛠️ Denormalization – Advanced denormalization techniques using incremental and refreshable materialized views.

🎉 This is one of the most requested blogs from our customers! If you are a Postgres user and want to learn how data modeling works in @ClickHouseDB, this blog should be a great summary. It also goes into quite a lot of technical detail. Please give it a read and I would love hear your feedback and questions! 🙌 Data modeling has been in my DNA for the past decade, and it was so much fun to be back at it after a few years! Thank you for driving the effort here, Lionel P., and for the collaboration, Christoph Wurm! 🙏

🚀🚀

📣Today, we are excited to announce the public beta of the Postgres CDC connector in ClickPipes! With this, you’ll be able to seamlessly replicate your Postgres databases to ClickHouse Cloud for blazing-fast analytics. No gates—we are making this feature available to everyone! clickhou.se/4gKWxjh 🎉 The Private Preview received an overwhelming response. We are grateful to customers like @Vapi_AI, @syntagecom, @neondatabase, @unifygtm, @SpotOn, @useblacksmith and many others who are using Postgres CDC to ClickHouse in production at scale and have shared valuable feedback. 🙏 🚀 Over the past few months, the team has shipped critical features, including User facing alerts, OpenAPI support, built-in monitoring of your source Postgres databases and more to provide an enterprise-ready experience. As a reference here is the landing page showcasing customer usage, benefits, and more: clickhou.se/4k6mKLW

🎉 We are excited to announce PeerDB v0.24.0, our second release of the year! This release of PeerDB features code changes to improve performance, observability and fine tuning of replication workflows. ⚡ PeerDB now pulls from Postgres and pushes to @ClickHouseDB in parallel (async), by default 🔧 Introduced knobs to tune memory usage of PeerDB inserts to ClickHouse via chunking ✅ Improved validation for read replica Postgres instances below PG16 📊 Added collection of more metrics such as errors emitted and rows synced 🚀 Re-architected parallelized syncs to ClickHouse and single table sync performance Here go the full release notes: github.com/PeerDB-io/peer…

🚀Latest release of PeerDB for Postgres CDC! In case you missed it, PeerDB is fully integrated into ClickHouse Cloud and available in private preview through ClickPipes. You can sign up here: clickhouse.com/cloud/clickpip…

🚀We are excited to announce the first major release of PeerDB in 2025! This release focuses on enterprise-grade Postgres to @ClickHouseDB replication, with key features centered on improved performance and enhanced reliability for Postgres CDC. 📦 Granular formats sync of binary and hex data from Postgres to ClickHouse. 🚀 Improved throughput for CDC by avoiding reconnections to support enterprise workloads. 🔁 Improved performance by using multiple replicas for data ingestion in ClickHouse. ⏳ Drastically reduced slot growth on idle databases by frequent acking on PKMs. 📊 Added internal endpoints for additional telemetry, including bytes moved and so on. ✅ Added additional validation to detect connection poolers. 🗑️ Improved reliability for Resyncs by removing capture of soft-deleted rows. 🛠️ Richer datatype support, including UUID arrays and tstzrange. 🔧 Revamped retry logic to better identify actual errors. 🧪 Enhanced end-to-end tests for various cases, such as partitioned tables.

Loved the discussion on this hn post over the weekend on Postgres for everything. news.ycombinator.com/item?id=423476… TL;DR from quite a few comments: just don’t do it! I’m glad this is becoming mainstream, and people are realizing that Postgres for everything doesn’t work. Don’t get me wrong—I’m a huge Postgres proponent and have spent 10 years helping customers implement it. However, I’m a strong believer in using Postgres for what it’s designed for in first-place. Postgres was designed as a row-based OLTP database, with over 30 years of effort dedicated to making it robust for that use case.I know there are many extensions attempting to make Postgres support other use cases, such as analytics, queues, and more. Keep in mind that these extensions are relatively recent and aim to retrofit new capabilities onto a database primarily designed for transactional workloads. It’s like adding an F1 car engine to a Toyota Camry — will that work? Extensions also have many issues—they are *not fully Postgres-compatible*. In Citus, for example, we added support for the COPY command four years into the company, and chasing SQL coverage was a daily challenge for 10 years. Being unable to use the full capabilities of Postgres and having to work around numerous unsupported features defeats the purpose of being a Postgres extension. On the other hand, you have purpose-built alternatives like @ClickHouseDB , @SnowflakeDB for analytics, @Redisinc for caching, and @apachekafka for queues. These technologies have benefited from decades of development, laser-focused on supporting specific use cases. As a result, they are robust and highly efficient for their intended purposes. I often hear that these Postgres extensions are expanding the boundaries of what Postgres can do. While I partly agree, I also question the extent to which these boundaries are truly being expanded. In this era of AI, where data is growing exponentially, handling scale is critical for any technology. These boundaries will likely be broken very quickly. Take queues as an example: you have a purpose-built technology like Kafka or a Postgres extension that supports queues. For an early-stage startup, adopting a less optimized Postgres-based solution may (not a guarantee) save a few weeks of initial CapEx costs compared to using an optimized solution like Kafka. However, 6 to 12 months later, you may find yourself back to square one when the Postgres-based queue fails to scale. At that point, migrating to a purpose-built technology becomes an arduous task—your system has grown, and now it may take months of effort and a larger team to make the switch. Ultimately, this approach can cost more time and money than starting with a purpose-built solution from the beginning, which might have only required a few extra weeks of CapEx. I’ve seen this firsthand at Citus, where customers like Cloudflare and Heap eventually migrated to purpose-built databases like ClickHouse and SingleStore respectively. While these migrations happened a few years later, times have changed — data grows faster now, and the need for a purpose-built database arises much sooner. It’s also worth noting that Citus was an incredible piece of technology that required years of development before it could start making a real impact. TL;DR: Please think carefully before choosing the right technology as you scale. Cramming everything into Postgres might not be the best approach for scaling your business.

🎉 Love all the exciting product releases included in the @ClickHouseDB product roundup for AWS re:Invent. clickhouse.com/blog/reinvent-… The Private Preview of the Postgres CDC connector in ClickPipes was one of releases. Now, you can start replicating your Postgres data to ClickHouse in minutes and enable blazing-fast analytics. Postgres can be hosted anywhere—whether on-premises or in the cloud. This includes RDS Postgres, Azure Flexible Server, Google CloudSQL, Supabase, Neon, Crunchy, Tembo, and more. This was the first result of the @PeerDBInc acquisition that happened few months ago! 👉 Go ahead and sign up for the private preview to get early access to the product. You can experience how magical it is to integrate Postgres and ClickHouse together. 🙂 clickhouse.com/cloud/clickpip…

📣🎉We are excited to announce that the Postgres CDC connector for ClickPipes is now in Private Preview! clickhou.se/4137EzL With this connector, you can natively replicate your Postgres data to ClickHouse Cloud in just a few clicks for blazing-fast analytics, eliminating the need for external tools that are often expensive and slow. Key benefits include. 🚀 Blazing Fast Performance: Achieve 10x faster initial loads with replication latency as low as a few seconds. 🎯 Super Simple: Start replicating in just a few clicks and a few minutes. 🛠️ Native Features: Supports Postgres and ClickHouse native features such as schema changes, custom order keys, and more. 🔒 Enterprise-grade Security: All data is encrypted in transit, with support for features such as SSH tunneling and Private Link. 🌐 No Vendor Lock-in: Powered by open-source PeerDB This launch marks a significant milestone following the @PeerDBInc acquisition. Many customers, including @SpotOn, @vueling, DaisyChain, and others, are already leveraging this technology to continuously replicate their Postgres databases to ClickHouse for analytics.

This is the first result of the @PeerDBInc acquisition by @ClickHouseDB that happened a few months ago. I am super proud and humbled of what the team was able to achieve in just a few months. Now there is a native way to integrate your Postgres databases to ClickHouse in just a few clicks! I want to thank all our customers, including @SpotOn , DaisyChain, @vueling , Adora, and others, who have been running production workloads for several months. Without you, this integration wouldn’t have been possible! I am a strong believer that customers will find the maximum value using purpose-built databases, as they are designed and with full flexibility, rather than relying on alternatives that retrofit one database engine into another, compromising the full feature set of each. This has been our approach with Postgres and ClickHouse—Postgres for transactional (OLTP) workloads, ClickHouse for analytical (OLAP) workloads, and a seamless Postgres CDC to bring them closer, forming a powerful data stack. We are seeing this trend across various customers who are using this data stack to solve most of their data challenges efficiently. I love this testimonial from one of our customers, @woodhull from Daisychain, which concisely summarizes our approach and is a testament to what we are building. “Clickpipes helps us reliably keep data from Postgres in sync with Clickhouse, while we rapidly improve our product. Clickhouse and Postgres are the peanut butter and chocolate of databases (they go well together!) and Clickpipes makes it easy to build that architecture at a fast-moving startup” ClickHouse and Postgres are the peanut butter and chocolate of databases. We are very excited for what is in store for the future. The vision is very clear - making it magical for customers to use Postgres and ClickHouse together. We will continue driving towards that effort. More to the future! 🚀

🎉 We are thrilled to introduce version 0.18.0 of PeerDB OSS, featuring crucial improvements to the core engine along with big pushes to observability and usability. Here are a few highlights: 🛠️ Remove tables from a mirror 📊 Advanced Postgres Peer Metrics: LSN graph, enhanced stat activity info ⚡ Efficient WAL reading by flushing via PKMs 🛠️ Specify custom sorting keys for @ClickHouseDB mirrors ✅ Improved mirror validation for Postgres CDC to ClickHouse ⭐ Native OAuth based Supabase peer

Use Neon and @ClickHouseDB to get the best of both worlds: the flexibility of Postgres for transactions + the performance of ClickHouse for analytics With PeerDB, data streaming is simple 👇

It was a lot of fun writing this blog! @neondatabase users can now seamlessly stream their transactional data to @ClickHouseDB for fast analytics! 🐘🚀 @PeerDBInc

🎉 Very excited about this integration with @neondatabase! Neon customers can now seamlessly stream transactional data to ClickHouse for blazing-fast analytics using PeerDB. Postgres and ClickHouse side by side means more power! 💪 neon.tech/blog/postgres-…

@PeerDBInc @ClickHouseDB Love the spirit of @PeerDBInc !!

🛳 🎉 Team has been shipping at full throttle post the @ClickHouseDB acquisition of PeerDB! 🚀 Here’s our latest release, version 0.17.0, with key improvements in usability, security, and reliability for Postgres CDC: 🔍 Auto-inference of NULL columns in Postgres to Nullable on ClickHouse 🗝️ Custom order keys on ClickHouse while creating MIRRORs 🛠️ Table engine selection for MIRRORs 📦  Smaller batch ingestion to reduce memory usage ✅ Improved MIRROR validation 🔐 AWS KMS support for encrypting PEER and alerting credentials

Can't wait to dive into my favorite part of any talk—the demo! 🚀 Cooking up something exciting to showcase the power of Postgres CDC + @ClickHouseDB . Get a sneak peek into building state-of-the-art CDC for Postgres to ClickHouse. Looking forward to connecting with fellow data folks at the meetup! 🎉 See you there! 👋 #DataEngineering #ClickHouse #PostgreSQL