Martin Schoombee

@SQLGene @mim_djo

@MartinSchoombee @mim_djo No, Mim has a fair and valid point. There's a needle to thread with providing enough details to help consultants and customers but not so much that you give away the crown jewels. I don't envy MS. I'm just a guy getting pissy on Twitter.

WAIT, is v-ordering just a Microsoft thing and not a native databricks feature?

@mim_djo @SQLGene It's like saying you should drive a BMW instead of Mercedes. Why? What's the reason apart from personal preference? And saying "just because it's better" is not enough in my opinion.

@mim_djo @SQLGene Of course...I don't think that's the point Eugene is asking about though. As consultants we're trying to provide the best guidance for our customers, and "do it because MS says so" is not great guidance without proof.

@SQLGene @mim_djo Yeah, I think the best we can do is: It'll be a game changer if they get it right, but until then it's better to use import mode.

@MartinSchoombee @mim_djo I don't begrudge them for 1) wanting a competitive edge and 2) keeping the secret sauce from Vertipaq secret. Microsoft is a business, I respect that. I would just like a smidge more detail so I can provide accurate guidance.

@SQLGene @mim_djo Open source but not open source, welcome to the new world Eugene. Matter of time before somebody reverse engineers the algorithm though...I think in this case it's all about the Vertipaq engine and how to make it perform outside of using its own proprietary file format.

@MartinSchoombee @mim_djo "V-Order works by applying special sorting, row group distribution, dictionary encoding" You can't just say "special sorting" and expect me to be able to provide good data modeling guidance. Like, the advice for Vertipaq modeling probably applies but whoooooo knooows

@mim_djo @SQLGene Haha, I think you know what Eugene means...he wants to know the internals, not just how to apply it.

@MartinSchoombee @SQLGene it is well documented learn.microsoft.com/en-us/fabric/d…

@SQLGene @mim_djo Agreed, but I don't think they can do that without giving away some secrets. They ultimately want to give people a reason to use Fabric over any other Data Lake platform, and "Direct Lake" would be one of the few reasons.

@MartinSchoombee @mim_djo So I guess logically they'd try to implement a similar sort of multi sort and encoding. BUT I STILL WANT THAT DOCUMENTED!

@SQLGene @mim_djo I think this is fundamentally the reason for v-order. They couldn't get the Vertipaq engine to perform adequately on Delta Lake with the standard compression & sorting.

@MartinSchoombee @mim_djo It's getting transcoded to Vertipaq but retains whatever sort was applied by V-Order.

@MartyCatherall Thank you sir!

Watching a fantastic #PASS session from @MartinSchoombee on using OAuth API's with data factory - Nice work Martin

@SQLGene @mim_djo And/or they were able to adapt the Vertipaq engine to read Delta Lake files with some adjustments. 2/2

@SQLGene @mim_djo My guess, and a very uneducated one for that matter, is that the MS claim that suggests they can get similar performance from direct lake means: By using their own sorting and compression (v-order), they get very close to the internal Vertipaq file format. 1/2

@FerrariAlberto Gets me every time :-)

Every single screenshot... DAX is soooo simple, compared to aligning a relationship!

@bob_duffy Hard "no" for me :-/

@erikdarlingdata Cupid...you don't have to worry about him this month.

Is there anything I don’t need to be aware of this month?

Do it for your country 🇿🇦 #proudlysouthafrican How to Braai: Starting the Fire tinyurl.com/c586cfjs A feast of culture, braai and fine wine tinyurl.com/4ec6r7uz The evolution of SA’s Heritage Day tinyurl.com/v2ykczv5 #desmondtutu #southafrica #ubuntu #28twelve

@GFritchey Describe "proper coffee" 😋

Americano is still not coffee. Fight me. Or, better not. I haven't had proper coffee. Things could get ugly.

#28twelve #powerbi #analytics in #southafrica

There are some valuable lessons here for #data analysts and #businessintelligence professionals. #analytics #cloud #digitaltransformation #28twelve

📝Blog Post: Rate my dashboard martinschoombee.com/2024/08/21/rat…

@AndyLeonard Meant to say for Basket A items it's probably between 10% and 15%.

@AndyLeonard All different, depending on the market basket. I think for the top 100 items people buy (usually Basket A) it's probably that, but others it could be as high as 100%.

What's the profit margin on groceries?