Samir Dutta

x.com/i/article/2069…

At @farsight_ai, we got tired of people explaining their workflows to AI - what they should look like, what they should include, how things are connected. Today we’re launching Freeform to fix that: SOTA, simultaneous large-scale generation across PowerPoint, Excel, and Word. Creating 100+ page slide decks, multi-sheet interconnected excel models, and robust and lengthy narrative documents from scratch is a thing of the past now. With Freeform, you can input a single prompt, and our system rationalizes across all of your data to understand not only what information is needed to execute something, but also what set of materials and outputs are required to complete it end-to-end. Freeform is already taking real, meaningful work off our customers’ plates. It understands how teams actually work, why they do things the way they do, and matches their way of doing things exactly. This is the first solution in finance that completely removes the mental burden of using AI, allowing it to do the work completely for you, exactly how you would do it. This fundamentally changes how financial work gets done. Extremely proud of what the team has built here. Comment “Freeform” to get a demo.

💀Prompt engineering is a painful necessity - we decided to automate it 🚀 👉Methodology Analysis Article:medium.com/p/45a7bea3d895 👉GitHub:github.com/farsight-ai/fa… 👉Docs: api.farsight-ai.com/farsight-opro 👉Original Paper: arxiv.org/pdf/2309.03409… "Large Language Models as Optimizers" proposed using LLMs as optimizers to identify prompts that maximize task accuracy. We've open-sourced our toolkit implementing this methodology as well as analyzed how it generalizes to optimizing via GPT-3.5-Turbo. Perfect prompts for your LLMs with just one line of code, in much less time.

🚨Open-source models are now beating GPT-3.5-turbo in the financial domain. Full post here: @kunal_79217/an-llm-benchmark-for-financial-document-question-answering-e63e9d2bda25" target="_blank" rel="nofollow noopener">medium.com/@kunal_79217/a… At Farsight, we created an automated benchmark to evaluate LLMs’ question-answering abilities on company financials. We find: 🤗 Open-source model SOLAR-10.7B now eclipses GPT-3.5 ❌ LLMs have a 20-50% error rate, even when provided gold contexts 🤖 Calculations remain a major weakness for LLMs (GPT-4 has >40% error)

A Stanford Lab released a 32k context length embedding model last week. Is it the real deal? ⚠️TLDR; No - embeddings saturate their information capacity at around 300-500 tokens. Break down on why below. Model: huggingface.co/togethercomput… Disclaimer: All credits for these insights go to @Nils_Reimers – I summarize a LinkedIn discussion of his here. - Embedding models optimistically store 1 byte per dimension (768 dim = 768 bytes). With optimal text compression (3.5x loss-free), this means 2688 bytes (768 *3.5) of text can compress into a 768 dim embedding. - The average LLM token contains 5 characters, and each character is ~1 byte. So, a 768 dim embedding can effectively encode ~540 tokens (2688/5) before the information storage quality drops. Well below 32k! - Increasing the dimensionality of embeddings isn’t a solution either. Embedding models struggle to encode > 1 topic per embedding effectively. Longer text > more topics > worse embeddings. - As a result, truncation and chunking are still the best strategies for longer contexts. Extra credit: Images differ, as they contain a ton of redundancies. They can be compressed at a much higher rate without noticeable difference. But, like text, more complex images with higher information content can also eventually saturate the information capacity of their embeddings. Nils has an upcoming benchmark and blog post delving into this material – stay tuned to him for more detail on this subject.