Victor G. Lesau
1.7K posts
@VictorLesau
Product #IoT Identity & smart home @AWS | ex- @CuePath (co-founder), @MicrochipTech @RogerrWireless @Techstars @StanfordEng @SFUResearch @McMasterEng
@ChatGPTapp @OpenAI @tszzl @emollick @voooooogel Wild result. gpt-4-turbo over the API produces (statistically significant) shorter completions when it "thinks" its December vs. when it thinks its May (as determined by the date in the system prompt). I took the same exact prompt over the API (a code completion task asking to implement a machine learning task without libraries). I created two system prompts, one that told the API it was May and another that it was December and then compared the distributions. For the May system prompt, mean = 4298 For the December system prompt, mean = 4086 N = 477 completions in each sample from May and December t-test p < 2.28e-07 To reproduce this you can just vary the date number in the system message. Would love to see if this reproduces for others.
OMG, the AI Winter Break Hypothesis may actually be true? There was some idle speculation that GPT-4 might perform worse in December because it "learned" to do less work over the holidays. Here is a statistically significant test showing that this may be true. LLMs are weird.🎅
