Tomiwa Adey
2.4K posts

Tomiwa Adey
@tomiwaAdey
Building and experimenting with 3D Agents


I can smell an AI generated frontend from 10 miles away.


Hi Jon, thank you for your interest! The project is essentially two parts: a low level physics engine that integrates a wide range of solvers for simulating materials with different physical properties, and a upper level LLM/VLM agent that uses the physics engine as tools to compose worlds and generate other modalities of data. The generative agent incorporates multiple generative modules handling different modalities in a modular way, and the high-level goal is instead of generating everything end-to-end, we generate them in a structured way and all of them collectively form a physical world :) Let me know if you want to dive deeper







async function createRealtimeSession(inStream, outEl, token) { const pc = new RTCPeerConnection(); pc.ontrack = e => outEl.srcObject = e.streams[0]; pc.addTrack(inStream.getTracks()[0]); const offer = await pc.createOffer(); await pc.setLocalDescription(offer); const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/sdp' }; const opts = { method: 'POST', body: offer.sdp, headers }; const resp = await fetch('api.openai.com/v1/realtime', opts); await pc.setRemoteDescription({ type: 'answer', sdp: await resp.text() }); return pc; }












