Marco Pedersoli

@PetarV_93 @gu_xiangming Also humans cannot be stochastic

can llms reliably roll the dice? 🎲 we shed new light on stochasticity limitations of llms, discussing some ways in which things can improve: tools, prngs, and 'just giving a random number to the model™' great work from @gu_xiangming while being a student researcher with us 🚀

@francoisfleuret I fully agree. And the space of not understandable reality it's probably much larger, but we cannot see it

Isn't the "intelligibility of reality" a facet of the anthropic principle? We do exist in a "slice" of reality (in space scale, time scale, and whatever informational projection) where the complexity of the signal meets that of the implementable computational devices?

@mihirp98 @pathak2206 @KaterinaFragiad @WuMengning54261 What about any-order autoregressive?

🚨 The era of infinite internet data is ending, So we ask: 👉 What’s the right generative modelling objective when data—not compute—is the bottleneck? TL;DR: ▶️Compute-constrained? Train Autoregressive models ▶️Data-constrained? Train Diffusion models Get ready for 🤿 1/n

@HeitorRapela @ICCVConference Are you still rev. 2 ? 😀

Finally, I was able to finish #ICCV reviews (@ICCVConference ) 😎 Now, towards the end of my Ph.D. I am much more confident about the quality of the reviews and feedback. I hope that the community can benefit from the comments.

@HeitorRapela @Mila_Quebec Well done. I am very interested to know about your findings!

Today we presented our project: The Role of Language in Knowledge Distillation from VLM, in the VLM course at @Mila_Quebec We had interesting finds on prompt learning, data aug, and rank loss for robust OOD distill. #AI #VLMs #ML #CV #KnowledgeDistillation

StarVector is a multimodal vision-language model for generating SVG (Scalable Vector Graphics) Performs image2SVG and text2SVG generation. 🤗 Gradio app is live! The video demonstrates how image generation is treated similarly to code generation task by StarVector MVLM! 🤩

StarVector is out on Hugging Face StarVector is a foundation model for generating Scalable Vector Graphics (SVG) code from images and text. It utilizes a Vision-Language Modeling architecture to understand both visual and textual inputs, enabling high-quality vectorization and text-guided SVG creation.

I’m excited to announce that 💫StarVector has been accepted at CVPR 2025! Over a year in the making, StarVector opens a new paradigm for Scalable Vector Graphics (SVG) generation by harnessing multimodal LLMs to generate SVG code that aesthetically mirrors input images and text. With this milestone, we’re also releasing StarVector on Hugging Face! 🥳🚀

🎉 Excited to introduce BigDocs! An open, transparent multimodal dataset designed for: 📄 Documents 🌐 Web content 🖥️ GUI understanding 👨‍💻 Code generation from images We’re also launching BigDocs-Bench, featuring 10 tasks to test models on: ➡️ Document, Web, GUI Visual reasoning ➡️ Converting images into JSON, Markdown, LaTeX, SVG, and more! 📜 Paper: arxiv.org/pdf/2412.04626 huggingface.co/papers/2412.04… 🌍 Website bigdocs.github.io

@jxmnop Recurrent transformer?

people should be doing multiple LLM forward passes with the same weights feels like you could get a lot more for same model size. and this is all differentiable, so training this way is trivial surely diffusion models aren't the only way to successfully reuse representations?

@RishavPramanik Thanks Rishav for your great work! It has been fun to work with you on this paper.

I'm happy to share about acceptance of our paper in #IJCNN2024 #WCCI2024 Masked Multi-Query Slot Attention for Unsupervised Object Discovery (arxiv.org/abs/2404.19654) Thanks to my co-authors (@marcopeddy). Looking forward to connecting with fellow researchers in Yokohama, Japan

@wellingmax Right to the point! How to avoid that?

It seems Global Warming is already fact. Indeed we should spend serious €€ to minimize & mitigate consequences. Instead we will wait until chaos ensues and then we will vote right wing power hungry opportunists with simple solutions to power. And we all know what happens next.

@ylecun @rmarcilhoo @erichammy @StevenLevy @DBahdanau @kchonyc @aidangomez But transformers can be trained in parallel because they do not have state vector, while lstm cannot. That's what is also making Mamba popular in these days. It can be trained in parallel because its state computation is linear over time, but its complexity for inference is fixed

@rmarcilhoo @erichammy @StevenLevy @DBahdanau @kchonyc @aidangomez Transformers are anything but cheap. I mean, the N^2 scaling law makes it somewhat impractical for high-dim inputs like video. That's why people have been working on local attention and other schemes.

This leaves out a bunch of prior innovations that *clearly* inspired the transformer authors. Chief among them is the whole idea of attention, which was popularized by "Neural Machine Translation by Jointly Learning to Align and Translate" by @DBahdanau, @kchonyc, and Yoshua Bengio posted in September 2014. This is the paper that started the attention craze: arxiv.org/abs/1409.0473 Self-attention is a clever trick which uses similarities between all pairs of inputs. This makes the network care about relationships between inputs, independently of their order (permutations equivariance). That's the real contribution of the 2017 transformer paper. But what really boosted the craze was the application of Self-Supervised Learning to transformers, triggered by the 2018 BERT paper, also from Google: arxiv.org/abs/1810.04805

@_akhaliq Very impressive!

Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a Single GPU Recent advances in large language models have brought immense value to the world, with their superior capabilities stemming from the massive number of parameters they utilize. However, even the GPUs with the highest memory capacities, currently peaking at 80GB, are far from sufficient to accommodate these vast parameters and their associated optimizer states when conducting stochastic gradient descent-based optimization. One approach to hosting such huge models is to aggregate device memory from many GPUs. However, this approach introduces prohibitive costs for most academic researchers, who always have a limited budget for many high-end GPU servers. In this paper, we focus on huge model fine-tuning on a single, even low-end, GPU in a commodity server, which is accessible to most AI researchers. In such a scenario, the state-of-the-art work ZeRO-Infinity suffers from two severe issues when running in a commodity server: 1) low GPU utilization due to inefficient swapping, and 2) limited trainable model size due to CPU memory capacity. The underlying reason is that ZeRO-Infinity is optimized for running on high-end GPU servers. To this end, we present Fuyou, a low-cost training framework that enables efficient 100B huge model fine-tuning on a low-end server with a low-end GPU and limited CPU memory capacity. The key idea is to add the SSD-CPU communication as an optimization dimension and thus carefully co-optimize computation and data swapping from a systematic approach to maximize GPU utilization. The experimental results show that 1) Fuyou is able to fine-tune 175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model, Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves 45 TFLOPS.

@_akhaliq I really think this is the way to go for LLM

Microsoft presents The Era of 1-bit LLMs All Large Language Models are in 1.58 Bits Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption. More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for training new generations of LLMs that are both high-performance and cost-effective. Furthermore, it enables a new computation paradigm and opens the door for designing specific hardware optimized for 1-bit LLMs.

@_akhaliq Isn't this just ensembling?

Tencent presents More Agents Is All You Need paper page: huggingface.co/papers/2402.05… find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence.

Thanks @_akhaliq for sharing our work! We introduce StarVector💫 a Large Language and Vision Model for generating SVG code, a new alternative to image vectorization! arxiv.org/abs/2312.11556 w/ @shubhamag1992, @ILaradji, @prlz77, @dvazquezcv, @chrisjpal and @marcopeddy 🧵👇

@saypramoun @IsmailBenAyed1 @dvazquezcv @ILaradji Thanks Saypra, it has been a pleasure to work with you!

@tunguz It does not look good for Europe :(

This is crazy.

@phillip_isola Yes, but it is using millions of images used by the pre-trained Imagen model.

This looks like one of those results that marks a phase transition in science: for years people have anticipated that synthetic data would eventually outperform / boost real, but an imagenet scale result has been elusive. Finally models are good enough that it works!