Mark Lyons

@andrewlamb1111 Thanks for sharing!

Here are the slides and recordings from our Boston DataFusion Meetup in September: Youtube: youtu.be/wCAud478Dg8 Slides (pdf): drive.google.com/file/d/18KGH_w…

@nikunj Yup

Spent 18 months trying to find what's coming beyond chat, here are some emerging patterns..

@cedar_db How is tpc-ds 1tb?

Have you ever wondered why existing database systems focus on either analytical or transactional performance? Learn why this is the case and how a hybrid storage engine can deliver high performance for combined workloads: cedardb.com/blog/colibri/

@vanlightly Cool

I'm working on a set of blog posts that compare the internals of Apache Iceberg, Delta Lake, Apache Hudi and Apache Paimon. No benchmarking, no judgments etc, just a comparison of internal mechanics.

True:

@petereliaskraft +1 really cool!

Firecracker is an incredibly cool piece of technology. Built by AWS and open-sourced, it's essentially a virtual machine monitor that tries to be as lightweight as possible, providing the minimal OS functionality most apps need to run (particularly network and file I/O) and passing through much of the implementation to the host OS. At DBOS, we use Firecracker microVMs to serverlessly host user applications. We really like them because they're fast to start up and don't require many resources, but provide the high level of isolation and security our users need. The AWS team that built Firecracker wrote a great paper about it--highly recommend checking it out if you want to learn more.

Microsecond-accurate time is now available in EC2 US East. So many cool things this makes possible: aws.amazon.com/about-aws/what…

@FintechKristen @BrightHorizons @reshmasaujani Yup. Crazy.

Public service announcement: two children in daycare at @BrightHorizons in Cambridge, MA costs $95,400/year. This is after-tax money (ie about $130k in income would be needed to afford this). Shame on this country. Cc @reshmasaujani

@JoshuaSteinman I’ve been working on measuring credibility & expertise via Proof of Research (proof of work concept) any interest in discussing.

Request for Startup: Batting average for public personae and organizations, preferably open and auditable. Perhaps an open database linking individuals to predictions, and enabling a sort of “Rotten Tomatoes” style rating for accuracy of both predictions AND overall accuracy.

Anyone looking for a new SA opportunity DM me and I can intro you to Roger Frey! (Great team & Roger is fantastic!!) lnkd.in/edKZsu-b

Verifying myself: I am markclyons on Keybase.io. 2RdVlnBARFNGHkBQEWYYppwhlr0zvyetUhBV / keybase.io/markclyons/sig…

@mim_djo @teej_m They deff are compressing and encoding the data and query execution as much as possible without materializing.

@teej_m something bother me and can't explain it, the only rational explanation for snowflake performance, they may be operating directly on compressed data or some shit like this.

@thetinot @mim_djo I believe for a fair compare you need to generate a net new tpch or ds data set to my comment the other day. There’s so much possibly fishy business w the data set already generated by snowflake

@mim_djo You're also using their dataset, which is optimized to heck and potentially extra cached. Not a fair analysis methinks

1/ Querying 40 GB of data from #duckdb, first try reading directly from cloud storage, the throughput is so slow, it hurt, after 10 minutes, get OOM for Query 18

@mim_djo Was it newly generated data set or a dataset they already created?

you think you have a basic understanding of OLAP database, then you run TPCH-SF100 ( that's 600 M rows) on #Snowflakedb using the smallest size, this is just wild !!! 102 second , I have no idea what they are doing !!!

TPCH-SF30 ; 180 million rows #AZURE D16DS_V5; 16 Cores, 64 GB RAM #Databricks Photon 41 S #DuckDB : 43 second Query Parquet files from the VM SSD, no Azure storage involved Databricks Software cost (not hardware) 4.4 $/Hour github.com/djouallah/Test…

@KyleJWeller Nice! 👍

We raised a $25M Series A to propel us forward in our mission to transform #dataanalytics: onehouse.ai/blog/announcin… 1yr ago we announced our company. We now doubled our team, built our product, and landed our first customers in production. Onwards! #datalakehouse #apachehudi

@DavidAMaier @neondatabase Very cool - Checkout @projectnessie for Lakehouse branching!

Wowowo, @neondatabase.. You are telling me you've built a database that allows me to just branch off my production data at any time in the past and use it for testing/debugging/development? Thats way too cool.

Join @dremio’s Tech advocacy & Eng team for the very first installment of the @ApacheIceberg Office Hours 📆 🚀 We will kick-off with a brief presentation on Copy-on-Write Vs Merge-on-Read strategies, followed up by Q&A on anything Iceberg related. When: December 7th, 12 PM

@thetinot 👍

Who is coming to re:invent???

Reminder, if you want to learn more about Apache Iceberg I have loads of resources plus a video series all curated in this article. -> dremio.com/subsurface/apa… #BigData #DataLake #DataLakehouse