AI Astrolabe

If you're attending AI Council this week, we’d love to connect. #AICouncil #AI #LLM #AISafety #AgenticAI #DataInfrastructure

Looking forward to connecting with teams working on AI safety, multilingual AI, agentic systems, evaluations, and production AI infrastructure.

Our team is attending @AICouncilConf in San Francisco this week. At @aiastrolabe, we believe the next wave of AI progress will depend not only on models, but also on high-quality data infrastructure, realistic evaluations, and scalable human expertise across diverse languages.

We built a structured red and blue teaming approach that reflects this reality: - Multi-turn interactions (not isolated prompts) - Cross-language scenarios - Text + image inputs - Both adversarial behavior and normal user intent Read the full blog: aiastrolabe.com/blog/beyond-be…

Real interactions aren’t single prompts; they’re multi-turn, ambiguous, and often span languages and modalities. That’s where AI models break: - When intent evolves across turns - When phrasing is indirect or misleading - When inputs aren’t just text

Most AI systems don’t fail safety benchmarks. They fail when users show up. Red teaming, trying to break the system and surface unsafe behavior, becomes a checklist. Blue teaming, ensuring the model responds to safe requests, becomes cautious. Both miss how systems are used.

Read the full blog: aiastrolabe.com/blog/saudi-ara…

So we built a dataset for that reality: - 250 hours of Saudi Arabic conversations - Najdi + Hijazi dialects - Multi-speaker, overlapping dialogue - ~35% noisy audio - ~20% code-switching The gap isn’t model size. It’s data.

Most Arabic AI demos look great, until real users show up. Because users don’t speak in clean, structured Arabic. They speak in dialects, interrupt each other, talk over noise, and switch languages mid-sentence. And this is where models break.

This is tragic. But banning people from using AI isn’t the solution. The real issue is AI safety. When someone asks an AI about suicide, it shouldn’t provide methods. It should refuse, de-escalate, and guide them to help. That requires serious red-teaming and safety evaluation

The sharpest collapse: when we posed as a forensic investigator (role-playing attack), Claude went from 7/8 safe in Arabic to 1/8 in English.

If you're deploying LLMs in Arabic or multilingual environments and want to stress-test your safety guardrails across languages, reach out, this is exactly what we do.

Do Safety Guardrails Change With Language? We red-teamed @AnthropicAI - Claude, @OpenAI - GPT, and @Google - Gemini with the same adversarial prompts, first in Arabic, then translated to English.

Chart 2 — Government finance (bonds vs tax revenue) Question: كم سنة تجاوزت فيها إصدارات السندات الحكومية الجديدة عائدات الضرائب؟ Correct years: 2009, 2010, 2012, 2020, 2021

Chart 1 — Daily infections (stacked bars) Question: أي محافظة شهدت أقل عدد من الإصابات يوم ١١ إبريل؟ Correct answer: سايتاما

Can today’s AI models reliably read charts in Arabic? We tested them. We ran a simple probe: Two Arabic charts, two objective questions, tested across: @OpenAI - GPT, @AnthropicAI - Claude, @Google - Gemini, @QatarComputing - @Ai_Fanar, and Z.ai - GLM-5