I have been doing a lot of interviews recently, and have asked LLMs to try to clean up the transcripts — no substantive changes, just removing filler words and adding paragraph breaks to make them more readable. But every time I've asked the models have hallucinated HUGE chunks of the transcripts — just totally inventing sections that don't exist. This is happening with both Claude and ChatGPT, with all sorts of prompts telling it not to do this. Just completely unreliable!

The issue seems to be that the model truncates the original transcript, so just skips over the middle bit (even when told repeatedly to read the whole thing). These aren't particularly long transcripts though; well within the context limit.

@ShakeelHashim this is a prompting issue

@ShakeelHashim I'm using Gemini 2.5 Pro (aistudio) to transcribe audio and to clean up the transcripts in both English and Russian it's almost perfect, but for audio, you've got to cut the files into chunks of less than 30-40 minutes, otherwise it all goes a bit pear-shaped

@ShakeelHashim Same!! And it feels like it’s gotten worse recently!

@ShakeelHashim it's not been an issue with me on Gemini at least not an issue I notice 🤔

@ShakeelHashim Gemini 2.5 Pro should solve your problem, I think - I’ve been super impressed by its command of long context

@ShakeelHashim I've used granola, which has worked pretty well

@ShakeelHashim Similarly, I've noticed that when I asked LLMs to translate a large chunk of Spanish text (e.g. a rental contract) it often summarizes it instead of fully translating everything.

@ShakeelHashim An obviously useful application that requires significant workarounds just to squeeze out a minor benefit to efficiency. Remind me again why we're worried about super intelligence? 🤷‍♂️

@ShakeelHashim I've been doing this for cleaning up formatting in verbal transcripts. The practical limit seems to be a few paragraphs, and even then it fails a few percent of the time and I need to have a detector that notices missing or added words and reprompts the model.

@ShakeelHashim Gemini 2.5 Pro on AI Studio with temperature 0.7 works pretty well for us.