Nico Perony

Does 3D generation always have to be either slow or complex and data-hungry?🤔 We don’t think so! With Geometry Image Diffusion, we’re all about reusing (and recycling ♻️) what already works — making it faster and easier by reducing complexity and data needs 🚀(1/10)

@VWGAMEDEV @unity blog.unity.com/engine-platfor… The entire AI team wrote this blog post. It doesn't get much more official than this Valentin

@nicolasperony @unity Well, We have also asked several times where the training data comes from. Why have we still not received a clear and official answer?

You asked, we listened 🙏. All of the Muse AI magic can now be accessed directly from within the @unity editor: a context-aware assistant (soon agent!), humanoid animations, LLM-powered behaviour trees, and more. Amazing collaboration between R&D and product teams ✨.

Combined with #unity3d real-time ray tracing, the final environmental atmosphere is presented on the @convaitech project. #gamedevs #RealTime #VFX #shader #madewithunity #Convai

We’re excited to introduce our group's new research paper, “Collaborative Control for Normal-Conditioned PBR Image Generation”, in which we tackle high quality single-view PBR materials! 🧵 arxiv: arxiv.org/abs/2402.05919 project page (with live demo!): unity-research.github.io/holo-gen/

The AI fun continues ✨ You can use Unity Sentis in open beta to build AI-powered experiences in the Unity runtime. Sentis reduces the complexity of importing and running AI models and enables creating lifelike NPCs and new interactions for richer gameplay. See it in the demo below that uses four chained AI models to create an open-ended dialogue and quest experience. Learn more about Sentis: on.unity.com/3szS7si #Unite2023 #UnitySentis

What if I told you there's a Neural 3D Reconstruction method that: * is mesh-based * 👀 renders at 120FPS 👀 * uses Neural Texture Fields NeuRas by @Waabi_ai is the new cool kid in town that nobody is paying attention to (yet), imo: 1/3

I don't have all the answers, but I can clarify a few things about @unity price changes, and hopefully address some of the (very valid) concerns. 1. The runtime fee only applies to devs whose game makes over 200k revenue in a year, AND over 200k lifetime installs. AND. Not OR.

@SylvioD @andytouch @andytouch I'd like to add that the promo video used a very early version of Muse which didn't have access to a lot of documentation content. The latest version shows this response instead, which I believe to be much better. Muse is currently in beta and continuously improving.

@andytouch Andy this is interesting as we are actually using chatgpt-4 APIs. Will look into it

Curious how similar Unity Muse and ChatGPT are; so I asked ChatGPT the same question about making a Match-3 (in Unity) and got a much better answer (very broad but the flow makes sense). From the video, the answer Muse gives is exactly how you should NOT make a Match-3 game. 😵

@lazlobonin We can't disclose specific details about training, but Muse knows about a number of Unity resources including manuals, tutorials, release notes, and scripting guides. The range of information available to Muse will keep growing as we roll out new updates.

So stoked today! We're unveiling products I've been working on with incredibly passionate teams across Unity: - Muse, an AI-driven vision for the future of content creation - AI-powered lifelike characters like Orb running on Sentis, our inference library: youtube.com/watch?v=Slw9hn…

The @huggingface space of TANGO is public: huggingface.co/spaces/declare… There are several limitations of the model e.g., it may not generate good audio for unseen concepts (e.g. singing) during training. To rectify this, we are training another version of TANGO on larger datasets.

3/ TL;DR of results (on the benchmarks we tested) is Classification: DINOv2 ≥ CLIP Dense tasks (segmentation, depth): DINOv2 > CLIP Retrieval: DINOv2 > CLIP

Intelligence expresses itself as play, imagination, exploration, creative analogy-making, passion, and very, very fast learning.

Base LLMs (non-finetuned) make very strong few-shot classifiers. Describe task in English, give few examples, read off the label probabilities on test example. No gradient-based optimization necessary. It brings a cannon to a knife fight but is fast, convenient, strong baseline.

🤔 I now revisit my assumptions about LLMs on a weekly basis. Lots of counterexamples too... But the good ones do make you wonder.

There are times and places for training your own models... With the release OpenAI's chatGPT API - coding is looking less like one of them. The human-eval pass@1 rate of ChatGPT is as good as the best Open Source model's pass@100 rate. And this is still just GPT 3.5...

I'm just really enjoying AGI fights on Twitter these days 🍿

My Transformers Catalog has become one of my most popular posts ever. Some of you told me that you turned into a pdf for easier reading. I thought I should make it into an arXiv preprint. Here you go: 60 Transformers in 36 pages 🤖 🎉 arxiv.org/abs/2302.07730

@mobileraj Not richer, but complementary. Audiovisual media captures a modality of human experience which is altogether absent from text. We are the product of our embedded & embodied multimodal perception. That's what Yann's driving at imo.

@nicolasperony Would you consider media as richer than text? If we believe VR could replace physical experiences, then it can all be vectorized.

Would like to unpack #11 further. Is he implying that the datasets avail only encompass a tiny fraction of human knowledge?