Leslie

@mike64_t hey is this open source? I was trying to recreate the same thing for a class project but I’d make the model play in a Minecraft manhunt instead

I think I can finally report some success training a quite accurate IDM capable of recovering keystrokes from Minecraft gameplay, even in quite PvP-heavy situations. At this point the model does not only know what keys are pressed to the extent reasonably discernible, it also knows how fast it is moving in 3D space at all times, even when knockback is mixing with the self-move impulse. Now, recovering keystrokes from normal external capture footage is just about impossible. E.g. W/A/S/D does exactly nothing during partial tick frames and jumping mid-air is also equally useless, so asking the model to recover key down states is inherently unreasoanble. Mouse deltas are also completely arbitrary units, as game mouse sensitivity introduces an arbitrary scale factor into the equation. The only good option is to think carefully about your model-environment contract, and only record "logical actions", not raw keystrokes. So here's a few unfortunate lessons I had to learn in roughly this order. - Choose good units. (bad: mouse deltas, good: delta radians [yes, you will need game-internal state]) - Capture from inside the main game loop and read the game fbo to get consistent frame-action pairing. Doing post-mortem pairing is hopeless. - Carefully define when you think keystrokes actually have an effect. (jump only works on ground, when flying or in water etc.) More subtle: The key may already be down, but no tick has happened yet to actually use the value. Hence: ignore Seperate gamestate into "fast and slow-moving" components. E.g. movement is likely tick based, camera rotation is very likely updated every frame in essentially every game ever. - Think about your frame-action correspondance contract (How old is the frame in relation to the inputs you capture? Will double or tripple buffering affect you?) Think about the game loop timeline, where you are sampling, how old the data you are reading is, and where the ticks are happening around you. Language models used to simply not have a model-environment contract, but even now with the model "living" in a designated harness, the contract still boils down to formatting, and tool implementation intrinsics. While also important, it is still quite a bit more obvious because the violations are in some way shape or form reflected as text you can actually see. - ffmpeg dropping frames cummulatively screws the model the further you get into the sequence because your targets are now shifted. If you can't encode the video in real-time, too bad. - Sodium has a frames in flight system different from vanilla Minecraft, which will also offset your targets from your frames. (there goes that data...) - Models are succeptible to latency. If there is too big of a delay between action and on-screen reflection, your performance degrades. At this point I realize ~100hours of gameplay is essentially no longer usable as a dataset. You can train on this data, but all you'll get is a mushy mess. However, some good news: - Making the model predict physics gamestate scalars helps the model generalize. For instantaneous events like jump, it's unreasonable to ask the model emit a short burst of jump=true at exactly the right time, however if you also predict your current y-velocity, the model has supervision signal for the "latent" from which that onground jump becomes apparent. Recovering x/z motion is also somewhat easier than unmixing it into plausible keystrokes for inertia-heavy player controller logic. - Regressing physics gamestate scalars also seems to make your dataset "bigger". While pure keystroke classification will overfit quickly, predicting exact physics gamestate scalars forces the model to generalize more and you can tolerate far more epochs before validation loss starts to stall out. This is the only reason why it was bearable to dump 100h+ of dataset hours and replace it with ~3 hours of gameplay after the 4th revision of the file format (yeah...) and somehow still have better performance. Now, you might be asking, "isn't this brittle?" and the answer is yesn't. Frame-action correspondance matters for training, but not so much during inference. So as long as you are sampling in roughly the same interval as your training data, you aren't violating any hard contract per-se. Somewhere around the frames ticks are happening, and during training you capture various tick-capture offset relations per random chance, so nothing is too obviously wrong here. HOWEVER, you will get screwed by gui scale, shaders, resource packs, "shit that recording is 1920x1040 because somebody doesn't know fullscreen exists" and other unfortunate edge cases of reality. But I suppose this is the role of dataset size. If all those "contract violations" that a youtube video has compared to the training data are addressed, I think this is a way to turn Youtube into a labeled dataset. I could never shake the feeling that VPT is a sound idea in practice, while never having been properly executed, and I think one reason why it hasn't is because that label boostrapping part is just a pain in the butt to get right. Now, what the player is doing is of course not the only label you can extract from video, but it has to be one of the targets predicted during pretraining to "align" the pretraining objective. Some notes on the video here, the colored dots on the analog visualizer are the ground truth, while the gray dot is the model prediction. Green means correct prediction, red means incorrect prediction at that frame. Model P(key) reports how wrong the prediction is from green (0.0) to red (1.0). You will also notice that during periods of rapid slow down, left and right actions become close to irrecoverable, because there is just that little motion. And some jump actions are not predicted correctly because I got the detection condition for jump events wrong... (duh) LMB/RMB for other than sustained events (like item-consume and block break) also seem to be hopelessly irrecoverable for now. Swing was supposed to do the same thing as motion y did for jump, but its too well behaved as an increasing counter. Maybe partial-tick interpolated values work better (v5 file format then... ugh..)

@sama Let’s goooo

To celebrate 3 million weekly codex users, we are resetting usage limits. We will do this every million users up to 10 million. Happy building!

@FadelMegahed @dpatil Thank you I’ll definitely be on the lookout for the next opening

@lesprgm @dpatil Hi Leslie, they open multiple windows. I believe that the summer window closed but the fall window should open soon. I had students to “co-op” virtually during the semester.

I've been spending time across the US with college students & faculty. There's a theme: students can't get internships this summer. Not just a few students. LOTS of students. Doesn't matter what kind of school you're at. The pipeline that used to work — apply, interview, get placed — feels broken in a way I haven't seen since 2008. Back then I wrote something called "Hack Your Own Summer" — the idea that if nobody's going to hand you an internship, you should build your own. Make something. HAVE SOMETHING TO SHOW FOR IT!!!! I think it's time to do that again, but bigger. Here's what I'm thinking: What if we stood up a program — free, open, student-led — where students who don't have internships this summer team up and build real projects? Real enough to put on a resume. Real enough to demo. Think of it like a summer-long hackathon or a YC-esque kind of thing with structure: teams, mentors, weekly speakers, and a demo day at the end. I don't have it all figured out. That's the point of this post. I NEED HELP!! If you're a student without an internship this summer — what would make something like this worth your time? I think this would be best if run by a bunch of college kids. Would you want to do that? If you're a professional who'd volunteer a few hours to mentor or coach — what would you need to say yes? If you've run something like this before — what worked and what didn't? Drop your ideas below. I'm reading everything. And if you're a student who wants to help build this from day one — LMK!

@FadelMegahed @dpatil I currently go to school in Ohio and just found out about this does this mean I am not eligible since the window has closed now that we are in April? I think it ended in March

@dpatil I think this is a great idea. The selection and teaming up students would likely be a logistical nightmare. That being said, I would be happy to help. In Ohio, we have this program: development.ohio.gov/business/third…. I am wondering how many similar programs exist!!

most browser agents are black boxes. you have no idea what they're doing until they break. your agent is a now lobster🦞 the OpenClaw community will understand.

i met him ronaldo🇵🇹

I love talking to random people that have dedicated their life to something. Whether it be investing, social media, sports, or stacking cups lol it’s consistently been the best way to inspire me creatively. If you’re an interesting person reply to this tweet! I’d love to chat :D