Ripsimé K. Bledsoe, Ph.D

How do we prepare students to WORK with AI rather than around it? Presenting this week at the TXST AI in Teaching & Learning Symposium: *AI-Resilience in OTD capstone writing *AI-Assisted qualitative analysis Sharing pedagogy+methodology! @txst @uiwcardinals

This is excellent because it is nuanced and realistic. You have hit on some of the most challenging aspects of ensuring assessment is truly giving you a snapshot of student learning starting from practice!

Number 2 is perhaps at the core of all of this not only because it establishes worth but also curriculum needs to afford intentional opportunities for retrieval and consolidation. Largely coverage dominates teaching and leaves little for much else!

One of the most balanced, deep dives of where we are headed w/ GenAI in education. An important point not talked about enough is we must understand/define learning and use it as both start and end point!

+1 for "context engineering" over "prompt engineering". People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step. Science because doing this right involves task descriptions and explanations, few shot examples, RAG, related (possibly multimodal) data, tools, state and history, compacting... Too little or of the wrong form and the LLM doesn't have the right context for optimal performance. Too much or too irrelevant and the LLM costs might go up and performance might come down. Doing this well is highly non-trivial. And art because of the guiding intuition around LLM psychology of people spirits. On top of context engineering itself, an LLM app has to: - break up problems just right into control flows - pack the context windows just right - dispatch calls to LLMs of the right kind and capability - handle generation-verification UIUX flows - a lot more - guardrails, security, evals, parallelism, prefetching, ... So context engineering is just one small piece of an emerging thick layer of non-trivial software that coordinates individual LLM calls (and a lot more) into full LLM apps. The term "ChatGPT wrapper" is tired and really, really wrong.

The zombie myth of learning styles that just won't go away! There is no evidence whatsoever that using or being taught with a """learning style""" produces better learning outcomes. Strategies, prior knowledge and the domain dictate far more. Stick to tech pls!

@karpathy Your list is why I have moved away from long threads and often start new conversations with different llms. I intentionally stopped the memory function when chatgpt first rolled it out because I found it stifled innovative responses and got stuck on a few knowledge pieces.

When working with LLMs I am used to starting "New Conversation" for each request. But there is also the polar opposite approach of keeping one giant conversation going forever. The standard approach can still choose to use a Memory tool to write things down in between conversations (e.g. ChatGPT does so), so the "One Thread" approach can be seen as the extreme special case of using memory always and for everything. The other day I've come across someone saying that their conversation with Grok (which was free to them at the time) has now grown way too long for them to switch to ChatGPT. i.e. it functions like a moat hah. LLMs are rapidly growing in the allowed maximum context length *in principle*, and it's clear that this might allow the LLM to have a lot more context and knowledge of you, but there are some caveats. Few of the major ones as an example: - Speed. A giant context window will cost more compute and will be slower. - Ability. Just because you can feed in all those tokens doesn't mean that they can also be manipulated effectively by the LLM's attention and its in-context-learning mechanism for problem solving (the simplest demonstration is the "needle in the haystack" eval). - Signal to noise. Too many tokens fighting for attention may *decrease* performance due to being too "distracting", diffusing attention too broadly and decreasing a signal to noise ratio in the features. - Data; i.e. train - test data mismatch. Most of the training data in the finetuning conversation is likely ~short. Indeed, a large fraction of it in academic datasets is often single-turn (one single question -> answer). One giant conversation forces the LLM into a new data distribution it hasn't seen that much of during training. This is in large part because... - Data labeling. Keep in mind that LLMs still primarily and quite fundamentally rely on human supervision. A human labeler (or an engineer) can understand a short conversation and write optimal responses or rank them, or inspect whether an LLM judge is getting things right. But things grind to a halt with giant conversations. Who is supposed to write or inspect an alleged "optimal response" for a conversation of a few hundred thousand tokens? Certainly, it's not clear if an LLM should have a "New Conversation" button at all in the long run. It feels a bit like an internal implementation detail that is surfaced to the user for developer convenience and for the time being. And that the right solution is a very well-implemented memory feature, along the lines of active, agentic context management. Something I haven't really seen at all so far. Anyway curious to poll if people have tried One Thread and what the word is.

What is learning? @P_A_Kirschner and I discuss why learning is inextricably linked with memory.

New randomized, controlled trial of students using GPT-4 as a tutor in Nigeria. 6 weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions. And it helped all students, especially girls who were initially behind

Have been sharing this research, along with attention residue, with my first year college students as I help them develop executive management skills. It has made a big difference as they understand why I build certain class norms for focused learning!

@HsunYuChan Thank you! Building on the great research you and many others have done on academic momentum!

@ripbledsoe Congratulations!

Thrilled to share my latest publication: From Setback to Success: A Restorative Momentum Model for Students Returning from Academic Dismissal. 🔸Key takeaway: Cultivating motivational habits and faculty support along with structured advising can help students bounce back stronger! Read here: doi.org/10.1177/009155… @SageEducation @CCollegeReview @TXHigherEdBoard @XueliWang1 @crisp_gloria @krwickersham @MikeFloresPHD

Absolutely! Features like this could remove the time consuming implementation for educators presented with endless guides and so many choosing not to allow use at all. It can also remove the need for external AI tools.

@emollick Absolutely. Code interpreter useful for qual research especially in ways that often limit its use. OpenAI's custom GPTs can be extremely useful for educators w ability to feed quality content and a means to scale up use in whole departments. But improvements much needed.

A side effect of OpenAI’s focus on AGI above all is that many of their most practical & useful tools are mostly left to rot. Code Interpreter is still the best AI analyst, but barely changed in a year. GPTs are still the best prompt sharing tool, but same. 💰 left on the table.

@BradleyKBusch I'm intrigued! I'm getting ready to do some professional development for faculty on the myth of learning styles and GenAI.

Putting the finishing touches to a blog to come out on Thursday. It explores the age old problem (learning styles) in a modern world (AI). I think it is my favourite blog I’ve ever wrote and I can’t wait to share it with everyone 😀🧠

Great reminder points in this article for those of us in the classroom as we approach the Fall semester!

Very interesting study on comparing RAG and long-context LLMs. Main findings: - long-context LLMs outperform RAG on average performance - RAG is significantly less expensive On top of this, they also propose Self-Route, leveraging self-reflection to route queries to RAG or LC. Report that Self-Route significantly reduces computational cost while maintaining comparable performance to LC. Interesting result: "On average, LC surpasses RAG by 7.6% for Gemini-1.5-Pro, 13.1% for GPT-4O, and 3.6% for GPT-3.5-Turbo. Noticeably, the performance gap is more significant for the more recent models (GPT-4O and Gemini-1.5-Pro) compared to GPT-3.5-Turbo, highlighting the exceptional long-context understanding capacity of the latest LLMs." Again, not sure why Claude was left out of the analysis. I would love to see that including other custom LLMs trained to perform better at RAG. I am not entirely convinced that long-context LLMs generally can outdo RAG systems today. But I think it's interesting to see a combination of the approaches which is something I've been advocating for recently.

Presenting at the Annual Teaching & Learning w/ AI conference in Orlando, FL (July 22-24) hosted by @UCFDigitalLearn. Pres topics: AI and study support & AI in the classroom w/ a HumanAIzing learning cycle.

Excited to present some of my recent work--Uncovering Student Insights: Using GenAI to Analyze Qualitative Feedback.

My recent article on how faculty can lead the way to shape GenAI in education especially as a student success tool. @TAMUSanAntonio @StanfordHAI @TingLiu_TAMUSA @HsunYuChan @krwickersham @XueliWang1 @britwag13 @VaSansone @BeverlyIrby