shaan☄️

Day 116 diving into ML. Read the NetHack Learning Environment (NLE) paper. Key takeaway: to get generalizing agents, create envs which are procedural + stochastic worlds and build memory + structured encoders into the policy from the start. also, use unseen seeds during eval (seeds reserved for eval) to expand on structured encoders from above, don't flatten the entire obs array, encode each modality into the right format like map data into a CNN, text based data into a language encoder, etc. feeding these into your policy allows it to learn richer representations like a human would, rather then sending a single giant vector

Day 115 diving into ML. Read DeepMind’s 2018 Quake III CTF paper. Notes: - Population based training: they trained a league of agents at once instead of one at time - Self-play with match making: generate an Elo score and match make between similar caliber agents so curriculum auto scales - Reward shaping for sparse wins: instead of only win/lose at the end, they learn a dense reward from in-game events (flags, pickups, progress) to make credit assignment tractable - Generalization via variation: procedurally generated maps + changing teammates/opponents forces transferable skills (this theme holds from XLand) Starting to see a similar recipe between these multi agent papers. Varied worlds + auto curricula baked into training via self play or skill based match making + learnable reward shaping

Day 114 diving into ML. Read OpenAI’s Hide-and-Seek paper (“Emergent Tool Use from Multi-Agent Autocurricula”). Notes: - Key takeaway: to create emergent complexity, hand authoring tasks is not the move, instead set up incentives + rich physics playground, and let agents generate curriculum for you - Pure competition created an autocurriculum: each side keeps inventing harder problems for the other, so difficulty self-scales - They didn't specify a reward for tool use , but tool use still emerged bc it was the shortest path to victory - Team rewards produced coordination + division of labor (agents specialize: builder, blocker, distractor) - Intrinsically motivated baselines (ie reward for exploration are not as effective as auto curriculum via competition (agent is forced to learn / explore new strategies to beat opponent) Lots of good, practical ideas in this paper regarding environment design as well. Worth the read.

Day 113 diving into ML. Read DeepMind’s XLand paper (“Open-Ended Learning Leads to Generally Capable Agents”). Key notes: - “General capability” came from an open-ended training loop that keeps making new challenges - The curriculum was auto-adjusting via constant feedback so that difficulty was always just right / training didn't stall - Multi-agent dynamics (coop + comp) act like a robustness regularizer - XLand is a procedurally-generated 3D task universe (not a fixed task list), so skills have to transfer across endless variations - Though, the policy adapts across variations within Xland, it doesn't necessarily transfer to all games (obvious) but worth noting - Cheap fine tuning for new games on this generalizing policy - World generator was the key here vs having a static environment, they could create endless scenarios

Day 112 diving into ML. Read the 2018 OpenAI "Learning Dexterous In-Hand Manipulation" paper. Key notes: - They trained a dexterous hand control policy entire in sim using PPO, then deployed it to the real robot (no irl demos) - Domain randomization across their sims was the real unlock - Randomization helped bridge gap between real world and sim, as there are many more variances IRL than could be modeled. Randomization helped capture some of that noise / variability - LSTM policy matters because randomizations persist across an episode - Main takeaway: for sim-to-real, creating a perfect simulator might be the wrong goal, instead train on distribution of worlds and memory enabled policies so robustness emerges by design

Day 111 diving into ML. Read the OG AlphaGo paper. Key notes: - They used a combo of SL + RL + MCTS - Policy net was good for approximating best moves, but value net helped with long-horizon position evaluation (less reliance on rollouts) - MCTS caches edge stats (N, Q) and uses policy priors to focus search - Essentially NNs give learned approximations, but search allows deeper strategic evaluation and decision making over longer time horizons (ie this move is less obvious right now but its tactically advantageous later) - Main takeaway: learn heuristics, then wrap them in search so we get consequence-aware decisions (not just “best-looking” moves)

Day 110 diving into ML. Exploring CUDA. Hunting for the next summit to climb.

What’s the best open alternative to OpenClaw right now? Doesn’t make sense to put all your data into it if it’s owned by OpenAI.

Day 107 diving into machine learning. Consolidation day. Writing about Flappy RL. Read a couple of blog posts / papers.

Day 106 diving into machine learning. I'm marking the Flappy Bird RL env and agent solved. Notable takeaways: 1. Solidifying curriculum into a single, repeatable training loop. 2. Switching from discrete phases in curriculum to a continuous ramp up of difficulty. 3. Holding the env steady and easy at beginning of training so the agent doesn't start with a moving target vs continuous ramp up from beginning 4. Simplifying reward design from 5 components to just +1 for passing pipe and -1 for dying. 5. Simplifying observations to just the base, objective data points (bird pos, dist from pipe, gap midpoint, gap height). 6. LR decay so we don't unlearn things as training goes on. These changes almost doubled performance of the agent. Simplicity is king in RL. Some things almost feel like philosophy (how would I teach a baby to do something). I'm putting together a blog post that covers my build process and takeaways. Stay tuned.