Ingar Haaland

How should one design survey experiments for maximal impact in economics? Here's my slide deck from a recent workshop in Uppsala. Thanks to everyone attending for making it a great experience @EconomicsUU drive.google.com/file/d/1yN4fQn…

@YanagizawaD @joachim_voth Yes, you spend tokens with Codex, but the token limits are so generous that the reviews have essentially have zero marginal costs!

@Ingar30 @joachim_voth @Ingar30 Your review system has zero marginal cost, conditional on having a paid ChatGPT subscription, no? (Seems like an important aspect for many people, in addition to the other reasons you mentioned)

Excited to release Reviewer: an open-source multi-agent reviewer for economics papers. Run it locally with Codex. It turns a PDF into a structured, high-quality report using a modular, highly customizable pipeline where every finding is traceable to an agent. GitHub in thread.🧵

@s_lauermann Depends on the paper, but perhaps 15-20 minutes in total.

@Ingar30 What is the usual run time for his? With multiple iterations of subagents, this may take quite a while.

@infornomics Nope, just me preferring Codex because of much more generous rate limits. If you ask Claude Code it will probably figure out how to migrate everything. I am excited about the possibility of using open-source tools though. Will try that in the future!

@Ingar30 Sweet, is there a reason this could not work with Claude code or even an open harness like pi?

@joachim_voth @joachim_voth if you have experiences with the other systems, try this one and let me know how it compares (or send me a PDF and I will generate one for you)

To be honest, I built it from scratch without looking at other reviewing systems as a learning exercise to get up to date on how to build a serious research workflow with Codex. What I think are legit advangates - extremely modular and easy to customize with Codex (built according to best practice Codex workflows) - it's also built around my preferences; if you want something different, just prompt Codex to change the report. For instance, you can keep all the agents but change what the editor prioritizes without risking breaking the system - open source and will further improve a lot with human feedback - extremely low frictions to get started. If you have Codex, should take you less than 5 minutes to clone my repo and install the Python packages before you can run your own review.

@jonasdovern Yes, *much* more generous token limits.

@Ingar30 Great. Will try. Is codex cli currently better in general compared to claude code in your experience? Or are there specific reasons for implementing this particular function in codex?

The reports are already strong, but the system is designed to improve over time with human feedback: better prompts, additional agents, improved PDF preprocessing, stronger validation, and more. Comments, bug reports, feature suggestions, and PRs are very welcome on GitHub.

The final Markdown report includes sections on: – executive summary – review configuration – prioritized cross-agent findings – revision priorities – additional findings – literature/novelty assessment – reference-integrity note – traceability info

To get started, clone the GitHub repo: github.com/Ingar30/review… The setup is straightforward and should only take a few minutes.

@anup_malani @SandroAmbuehl The publication lag is too large for this to be useful. If it's in your sub-field you've probably read the paper 4 years ago.

How many of us read the top 5 in its entirety each issue? Small % I guess. Perhaps even in our field. Also non-peer reviewed NBERs get lots of views. Not saying small attention span is not real. Just that it may not map to top 5 issues. My take is top 5 is a job allocation issue. We have far fewer tenure slots than economists. We need to allocate. Our screen is too 5. The stakes a really high, so people signal with ever more effort per paper, even when social returns to that effort are negative (opportunity cost of that time).

The core constraint is attention-everyone wants to read the best x% of work in their subfield. This constraint is unchanged, so the number of slots considered attention-worthy has stuck. To solve the issue, the attention constraint is key, e.g. more subfields or shorter papers...

@PriceTheorist I worked on some extensions and I can tell you it is wild!

@Ingar30 so cool! I tried to do the same with Claude and it refused

Finally got around to testing the "goal" function in Codex. Asked it to build a survey software similar to Qualtrics but optimized for use with Codex, and in less than 10 minutes it had one-shotted this. Pretty impressive!

The link to the repo is here, so try it out sooner rather than later: github.com/Ingar30/qualtr… It was initially created as a teaching exercise for my PhD course "Transforming Economic and Financial Research with AI" to illustrate a Codex workflow, but I hope it can be useful more generally!

It also produces slides that summarize the workflow and include all the relevant figures.

Do you run Qualtrics surveys? Then use my GitHub repo to automate the whole process with Codex! Tell Codex what you want to study, and it will create the survey, push to the survey to Qualtrics and generate synthetic responses, download and clean the data, and produce a set of slides summarizing the process.🧵with complete guided example!