Marco Tulio Ribeiro

Testing LLMs (and prompts) like we test software: towardsdatascience.com/testing-large-… TL;DR: (1) You should, (2) How to test: specific properties, evaluate these with LLMs (perception is easier than generation), (3) What to test: get the LLM to help you figure it out.

Microsoft open-sources a new AI library that connects to open-source GPTs, not just OpenAI. github.com/microsoft/guid…

@sean_lynch We're just writing stuff WE would want to use, and I guess we probably count as 'real developers' :)

It's mind blowing how well Microsoft now understands real developers in a way that none of rest of MAAG does. Look at github.com/microsoft/guid… - handlebars - python/jupyter/pip - tested against llama - references community projects There's zero BigCo not-invented-here taint

Also highly relevant: guidance from microsoft "Guidance programs allow you to interleave generation, prompting, and logical control" Also internally handles subtle but important tokenization-related issues, e.g. "token healing". github.com/microsoft/guid…

been reading the readme for github.com/microsoft/guid…, kind of galaxy brain tl;dr they made a whole prompt engineering language

Blog post: playing with Vicuna-13B, ChatGPT (3.5), MPT-7B-Chat on harder stuff @marcotcr/exploring-chatgpt-vs-open-source-models-on-slightly-harder-tasks-aa0395c31610" target="_blank" rel="nofollow noopener">medium.com/@marcotcr/expl… TL;DR: We think ChatGPT is still way ahead, but sometimes the extra control from open source models is worth it.

My intern is close to writing a paper, so I wrote her this blog post on writing (part 1 of 2): @marcotcr/writing-part-1-the-process-6bb92cb522eb" target="_blank" rel="nofollow noopener">medium.com/@marcotcr/writ…

I never tweet, but here is a blog post I wrote for an intern, may be useful for others too... Part 1: @marcotcr/coming-up-with-research-ideas-3032682e5852" target="_blank" rel="nofollow noopener">medium.com/@marcotcr/comi… Part 2: @marcotcr/organizing-and-evaluating-research-ideas-e137637b599e" target="_blank" rel="nofollow noopener">medium.com/@marcotcr/orga…

@jacswork My guess is you mean LIME : ). I don't know exactly what you mean, but we have follow up work coming out soon!

@BecomingDataSci Twitter is too hard : )

@BecomingDataSci Would love to hear what went wrong if you are willing to share in detail. Would you email me at my handle @gmail.com?

@BecomingDataSci I can share a few additional text or tabular examples if you like, but they all require specific datasets. We have a ton.

@BecomingDataSci Which ones didn't work? The first three should work, so please let me know if there are bugs =]

"Why Should I Trust You?" Explaining the Predictions of Any Classifier. Promo video: youtube.com/watch?v=hUnRCx… #kdd2016 @guestrin @sameer_

@fmailhot Heh, sorry for opaqueness of my replies. Twitter is not really my thing. Feel free to email me further questions or comments: )

@fmailhot Good point about opaque features. If the classifier uses stopwords, LIME should reflect it, so I don't think LIME should remove it

@fmailhot Probably don't need LIME for that though, if it's only 5 tokens. Things change in a longer sentence (not even a long document).

@fmailhot That is true (only 32 data points), but you may still want to tease out the contribution of each token. e.g 'I do not like that.'

@fmailhot It works with documents of any size. We also just added support for tabular (numerical + categorical) data. Maybe images soon.

Great work from @marcotcr , @sameer_ on explaining any machine learning model (20 newsgroup, deep net). homes.cs.washington.edu/~marcotcr/blog… @guestrin