Jaas AI

@svpino What about implementing monitoring tools that could look for the non-deterministic outcomes (ie Hallucinations) in parallel to the agent?

Every single solution to fight the non-deterministic nature of Large Language Models comes with trade-offs. There's no free lunch. Some people have suggested running the same process 3 times and choosing the most common answer. That would definitely improve reliability, but it will make the solution 3x slower. Anybody who's spent more than 10 minutes building software for a living understands why that non-deterministic nature is a big issue. If reliability were all that mattered, you could find 1,000 ways to improve it, but there's always more at stake.

@svpino Even better if you can have real-time monitoring for some AI apps to detect issues (i.e. hallucination before they happen!) - jaas-ai.net

The only thing worst than not having tests is having useless tests.

@svpino AI testing is important to ensure that your applications (including chatbots) behave as their should. There are some tools out there... check JaaS-AI.net.. ;)

These are some of the most mediocre people: The same folks who hate unit tests are now up in arms against writing evals for their LLM-based applications. If you are a solo developer or a startup building a prototype, you shouldn’t spend your time writing tests or evaluations. Instead, focus on iterating as fast as you can. But as soon as you are ready to go to production and scale your product, you gotta start writing automated tests and evals. Testing/evals are 10x more relevant when building agents and 100x more relevant when building critical workflows. Nobody has ever built anything of value without extensive eval coverage.