JJ Allaire

We are excited to announce Inspect Scout, a tool for in-depth analysis of AI agent transcripts: meridianlabs-ai.github.io/inspect_scout/ Scout lets you go beyond simple success/failure metrics to detect issues like misconfigured environments, refusals, and evaluation awareness using LLM-based or pattern-based scanners. Scout includes tools for developing scanners interactively, validating rubrics, and exploring scan results visually. We are especially appreciative of the feedback we got from @AISecurityInst, US CAISI, @METR_Evals, and @apolloaievals during the development of Scout. Blog post: aisi.gov.uk/blog/a-pipelin… Website: meridianlabs-ai.github.io/inspect_scout/

Excited to share details on two of our longest running and most effective safeguard collaborations, one with Anthropic and one with OpenAI. We've identified—and they've patched—a large number of vulnerabilities and together strengthened their safeguards. 🧵 1/6

We at @AISecurityInst worked with @OpenAI to test & improve Agent’s safeguards prior to release. A few notes on our experience🧵 1/4

To interpret AI benchmarks, we need to look at the data. Top-level numbers don't mean what you think: there may be broken tasks, unexpected behaviors, or near-misses. We're introducing Docent to accelerate analysis of AI agent transcripts. It can spot surprises in seconds. 🧵👇

Today, we're marking our anniversary by releasing InspectEvals – a new repo of high quality open-source evaluations for safety research. aisi.gov.uk/work/inspect-e… 1/2

@ClementDelangue @soundboy @IreneSolaiman @clefourrier @ClementDelangue and @clefourrier, would love to work with you on this (we already have native support for HF models and datasets). Will DM you to follow up.

@soundboy This is very cool, thanks for sharing openly! Wonder if there’s a way to integrate with hf.co/models to evaluate the million models there or to create a public leaderboard with results of the evals (ex: huggingface.co/collections/cl…) cc @IreneSolaiman @clefourrier

1/ Today the UK's AI Safety Institute is open sourcing our safety evaluations platform. We call it "Inspect": gov.uk/government/new…

Consider applying to UK AISI's technical Safeguard Analysis Team! Governments need a clear/SOTA understanding of how well safeguards work in frontier AI systems. Short 🧵 with some team info; deadline for this round 27/2. 1/6

The UK AI Safety Institute is hiring more technical staff! I believe AISI is one of the best places to do ML research/eng for the public good. We’re having impact at the scale of government, while moving at the pace of a startup (what more could you ask for?)

@hadleywickham @rstudio I am certainly even more grateful that you decided to join. I couldn't have imagined all that was possible and am so looking forward to the years ahead.

Today is my tenth year anniversary at @rstudio. It's hard to overstate what an amazing place RStudio is to work at and how much of a positive impact it's had on my life. I'm so grateful to @fly_upside_down for asking me to join the team, all those years ago.

Our biggest launch in years: nbdev2, now boosted with the power of @quarto_pub! Use @ProjectJupyter to build reliable and delightful software fast. A single notebook creates a python module, tests, @github Actions CI, @pypi/@anacondainc packages, & more fast.ai/2022/07/28/nbd…

To celebrate the launch of #nbdev v2 and @quarto_pub, I sat down with the CEO of @rstudio, JJ Allaire, to talk about software development, scientific publishing, #rstats, #Python, literate programming, and much more. I hope you enjoy it as much as I did. youtu.be/xxVVSxcjNQs

@benmarwick @Iza_Romanowska @CAA_Int @idhrenil @alex_brandsen @tombrughmans @electricarchaeo @ArchaeologistSP @xrubiocampillo @jwhpverhagen @lornarichardson @verdewek @CDWren @joeroe90 @er_crema @RachelOpitz @Iza_Romanowska We are currently working on adding Distill style features to Quarto (e.g. see quarto.org/docs/authoring…). We'll also be adding features for blogging and syndication soon.

@Iza_Romanowska @CAA_Int @idhrenil @alex_brandsen @tombrughmans @electricarchaeo @ArchaeologistSP @xrubiocampillo @jwhpverhagen @lornarichardson @verdewek @CDWren @joeroe90 @er_crema @RachelOpitz This sounds very exciting! quarto.org could be ideal for this, it's a scientific and technical publishing system built on Pandoc & markdown for both R & Python users, with Jupyter and Knitr support.

In these uncertain times @CAA_Int is going forward strong. The kind of output we produce as computational archaeologists is also changing and our publication model should change accordingly. I'll spearheading this transformation, so stay tuned!

@bernhardsson @eddelbuettel @JasonAizkalns @rstudio @bernhardsson Check out Jupytext (jupytext.readthedocs.io). You could use the percent format (#the-percent-format" target="_blank" rel="nofollow noopener">jupytext.readthedocs.io/en/latest/form…) which is supported by just about every Python IDE then convert to a notebook w/ Jupytext if/when necessary.

@eddelbuettel @JasonAizkalns @rstudio @fly_upside_down This looks quite cool but I'm trying to keep my examples runnable for anyone without installing any tooling, so I want to keep it as .py with inline docs

Is there a way to document Python scripts in a literate programming way? Stop me before I write some hacky parser to turn .py files into .rst so I can feed them into Sphinx

@EmilyRiederer @bernhardsson It's under development but definitely in a good place to use (quite a few folks are using it on real projects and we try to be extremely responsive when issues are reported)

@bernhardsson Quarto is a new tool in the spirit of RMarkdown but with Python and Julia also as first class languages quarto.org I’m not entirely clear how much it is still under dev tho

@leoferres @JuanUgaldeC @ZorzalErrante @Fcorowe Good news, this is being actively worked on in and around Jupyter core (see discourse.jupyter.org/t/inline-varia…). Once a clear standard emerges from this discussion we will support it.

@leoferres @JuanUgaldeC @ZorzalErrante @Fcorowe This is the current workaround: #jupyter" target="_blank" rel="nofollow noopener">quarto.org/docs/computati… (but that's obviously not very elegant)

Is there a #python implementation of something close to #RMarkdown, that let's me evaluate code in the document? I'm aware of #pweave, but it's not working out-of-the-box, and it seems to be abandonware...

@leoferres @JuanUgaldeC @ZorzalErrante @Fcorowe You can't currently run inline Python code but this is something we're hoping to get working soon.

@jannikbuhr @gvwilson @gvwilson We are definitely planning on a pretty robust extension mechanism (including custom shortcodes).

@bmwiernik @rstudio FYI have now changed the docs to use CSL rather than BibTeX: github.com/rstudio/distil…

@rstudio Is the citation engine still built on pandoc? If so, it might be good to encourage use of CSL YAML rather than BibTeX for entering bibliography data (#citations" target="_blank" rel="nofollow noopener">rstudio.github.io/distill/#citat…). CSL is pandoc’s native format, and converting BibTeX can yield incorrect citations (eg, for software)

Announcing Distill for R Markdown v1.0: A publishing format optimized for scientific and technical articles, websites, and blogs. blog.rstudio.com/2020/12/07/dis… #RMarkdown #RStats #RStudio #DataScience

@bmwiernik @rstudio Yes it's still based on pandoc so we could indeed change this example now. Good catch!