I have been doing a lot of interviews recently, and have asked LLMs to try to clean up the transcripts — no substantive changes, just removing filler words and adding paragraph breaks to make them more readable. But every time I've asked the models have hallucinated HUGE chunks of the transcripts — just totally inventing sections that don't exist. This is happening with both Claude and ChatGPT, with all sorts of prompts telling it not to do this. Just completely unreliable!

The issue seems to be that the model truncates the original transcript, so just skips over the middle bit (even when told repeatedly to read the whole thing). These aren't particularly long transcripts though; well within the context limit.

@ShakeelHashim For this kind of task, I've seen reliability trail off sharply between 20-50% of context window length. Keep your chunks below 20% of context length and it should be fine. If it isn't, send me an example prompt and transcript—happy to help figure it out.

@ShakeelHashim If you have lots of these, it's probably worth vibe-coding a script that splits files into ~1K word chunks, sends the tidy requests to a model API, then stitches them back together. That'd be the super safe way to do it. For ultra safe, add some automated checks.

@peterhartree I managed to fix it via a prompt (which Claude came up with). But thank you!