Andrew Snare

@jaceklaskowski I don’t have a query to share, sorry. I thought I saw on Spark 3.1 that with repartition() anywhere then the CustomShuffleReader stages seem to disappear. I assumed at the time that AQE was disabling itself. Next week I’ll be revisiting that job and have a closer look.

@asnare Hmmm...repartition is a wide transformation while AQE wants to be 'coalesce'-like, i.e. a narrow transformation. Could this be the case? Do you have a query plan to share?

Can spark.sql.shuffle.partitions be 'auto' in #Databricks? I haven't seen it in #ApacheSpark #SparkSQL ever. Just a number (which by default is 200 and should be changed to match your workloads). I had a call with a customer who showed me this but couldn't explain 😟

@jaceklaskowski Ooh. Nice find; thanks! I need to digest that a bit.

@asnare Dug it out from the links at stackoverflow.com/q/41359344/130… 😏 I also remember I asked this question on dev@spark and Key replied that this is what they came up with as a sensible default after their tests. Can't find this in the archives though.

@SwiftOnSecurity One of my most profound “digital” experiences ever was the day as an undergrad that we plugged the thin ethernet cable into the bench oscilloscope. We could see the signal, jaw-dropping how messy it was to the point of “how the fuck does it get this right?!?” … “Oh…” 😳

What you do not understand is that modern interconnects do not operate with 1=1 and 0=0. Shit they didn't work that way in the 8086. They are hyper-complex mathematical regimes that can compensate only up to a delicate point at a certain temperature. Functionality is a miracle.

Okay I have _LIVED_ this multiple times. Swap out the Ethernet patch cable. To the block and in the patch panel. Holy fuck you get told TCP checksums solve all this shit they don't. The most inexplicable things I could never figure out for anything rational. Just fucking solved.

@jaceklaskowski [1] Partitions that expand due to something like explode() basically break AQE, which is unfortunate, and if I recall trying to fix it in that area with repartition() just disables AQE entirely.

@jaceklaskowski I’ve started setting it to “large” values in the thousands: 1) apparently nobody knows the origin of 200 as a sensible default; 2) with Spark 3, AQE will normally[1] reduce it to something sensible; 3) the downside of too many partitions are nowhere near as bad as too few.

@gglanzani @Twitter @Twitterrific It’s hard to say; the macOS version seems to be unaffected. 🤷‍♂️

Did @Twitter just cut API access to @Twitterrific?

@SwiftOnSecurity Here’s some data I collected at a large client with SSO on everything internally. Also: hideous complexity rules, regular rotation, pwd managers blocked; <1hr into onboarding, trying to come up with a password, someone calls across the room “Just use […]”. So everyone does.

Note this is a choice some IT person made years ago and nobody realizes is optional, or something is actually broken and people just got used to it. Re-Auth could be entirely seamless if they wanted, or only enforced for certain apps. This is not intrinsic it could be seamless.

This map of Texas superimposed over Europe really drives home how small and insignificant Europe is compared to America. No wonder something like 97% of Scandinavians say they would prefer to live in the USA.

They all are.

🫣

Bellingcat Discord member Sepulco geolocated the video Elon Musk posted overnight, claiming to show a stalker, which he used to justify the banning of accounts that tracked flights. It's at 34.116175,-118.160635, not close to any airports. @34.1161302,-118.1607175,3a,75y,63.89h,75.56t/data=!3m6!1e1!3m4!1sUXpoDp_YWjnuhp9rnEylrQ!2e0!7i13312!8i6656" target="_blank" rel="nofollow noopener">google.com/maps/@34.11613…

It appears Elon Musk, the “free speech absolutist,” has shadowbanned an account which shared publicly available data regarding his private jet. trib.al/KMtyoqP

Anti-monopolism is the pathway to ecstasy

@ratkins Sometimes the phones all go batshit now as well like someone’s movie-hacked the country.

The first time I was in Amsterdam for the first Monday of the month nobody told me about the warning system test. They have klaxons on every street corner. I literally hid under my desk, I had no idea what was going on.

New Tesla ‘Memories’ Feature Displays Pictures From Driver’s Life On Dashboard Before Car Explodes bit.ly/3OQJ6C2

I remember the old saying, “Better to be thought a fool than to buy Twitter and remove all doubt.”

This whole Mastodon thing reeks of XMPP federation dreams however instead of a whole bunch of high quality high performing XMPP servers that can handle hundreds of thousands of requests a second with server 2 server we got 1 janky Mastodon Ruby implementation federating the globe

@jimseven @WIRED I have some ancient attic-found Cona makers that have for years been a mystery; felt way too niche to ever justify an Ultimate Guide (or even a mention). That syphon answer just… solved so many problems! Very excited to have another crack at them.

Delighted to have been a guest on @WIRED’s Tech Support series - here’s Coffee Support: James Hoffmann Answers Coffee Questions From Twitter | Tech Support | WIRED youtu.be/pjJqOgFyCxI