Jonathan K. Kummerfeld

Still accepting applications for this postdoc position in my lab at NYU! Applications due Feb 1. Links to posting and Interfolio application in thread.

Do you have ideas for the future of reading? Submit a 2-4 page paper to the CHI workshop I am co-organising! (deadline Feb 12) “Science and Technology for Augmenting Reading" chi-star-workshop.github.io

I'll also be at ACL presenting this, along with 2 papers on clinical coding from my group!

@DrorRotem @VeredShwartz @LChoshen @GabiStanovsky Expected to AC 4 papers per submission. In other words, a meta-review and a review are treated the same in the load calculation.

@jkkummerfeld @VeredShwartz @LChoshen @GabiStanovsky what load is that? Review load or AC load? or to be more clear - if I'm an AC and I submitted a paper, am I expected to be the AC of at least 4 papers, or, in addition to the AC load, also review 4 papers?

The ARR policy requiring a qualified author (3 main *CL papers) to review 4 papers per submission makes it impossible for junior faculty to submit papers. I have 9 students, only 2 are qualified reviewers. I will need to review 16 papers in this cycle to avoid desk rejects. 1/2

@NitCal @VeredShwartz @LChoshen @GabiStanovsky Best to put the ACs in the box, but we are trying to check them even if you don't (it gets complex so we have moved to a generous interpretation). For future cycles this will be clarified.

@jkkummerfeld @VeredShwartz @LChoshen @GabiStanovsky To avoid automatic desk rejection, if all submission authors are ACs, should we leave the 'reviewing volunteering' field empty or list the ACs' names? In either case, should we check the 'All qualified authors are already involved...' box?

@iamRezaSayar @VeredShwartz @yuvalmarton @ilyasut @GabiStanovsky Note - using an LLM like this would violate the ACL Policy on Publication Ethics (#Reviewing" target="_blank" rel="nofollow noopener">aclweb.org/adminwiki/inde…). If you want a range of other ideas for how to use AI / NLP to improve reviewing, here is a longer answer that I contributed to: arxiv.org/abs/2405.06563 :)

@VeredShwartz @yuvalmarton @ilyasut @GabiStanovsky intresting response! although unfortunately I'm not familiar with the training value work that you mentioned. Also, don't think about it as 'training', rather asking an LLM. one could even host a touring test of their own like what they've done here! 😜 journals.plos.org/mentalhealth/a…

@MoZaman412041 @VeredShwartz @srchvrs List them for all papers for this cycle. For future cycles we will work to provide guidance.

@jkkummerfeld @VeredShwartz @srchvrs @jkkummerfeld If we have more than four papers but only one qualified reviewer—and we are unable to find any other non-author to review—can we list the qualified reviewer for all the papers? Or can we only assign them to four papers and request an exemption for the others?

@tallinzen @yuvalmarton @VeredShwartz ARR tried spreading the load evenly by introducing a minimum availability and is now trying a form of proportionality. Both cause frustration. At the end of the day we are all researchers with insufficient time, but there is no magic solution :/

@tallinzen @yuvalmarton @VeredShwartz We are certainly in a bind - it's a zero-sum game. Every paper needs 4 reviews (3 reviews + meta-review). So that's a lot of work that someone has to do, and as a community we quite reasonably don't let just anyone review (so quite a few papers have no one qualified)

@VeredShwartz @srchvrs Yeah, we'll be working on refinements to the form for future cycles

@jkkummerfeld @srchvrs Thanks! OpenReview wouldn't let me add non-authors. I put their names in the justification and filled out the other form even though we didn't get the link to it for the particular submission. I'm glad to hear it's not fully automated (and sorry for the amount of work it creates)

@zehavoc @VeredShwartz @srchvrs This is a very fair concern and something that will be discussed further I expect (note, I am not the decision maker on policy)

@jkkummerfeld @VeredShwartz @srchvrs what would be the incentive for non-authors to volunteers reviewing for their colleague's work? Everyone is drowned under work, why would they sacrifice their time?

@tallinzen @VeredShwartz As a member of the team, we are your colleagues and trying to make the thing work - feedback is heard, though sometimes the speed at which decisions need to be made leads to sub-optimal communication

@tallinzen @VeredShwartz The decisions are made by the EiCs. Consultation about ARR happens with the community regularly - business meetings, via email to the team, and feedback in the large surveys that have been run.

@MoZaman412041 @sarvk @VeredShwartz @CatarinaBelm Not at this time, but these requirements are likely to be reviewed for future cycles.

@sarvk @VeredShwartz @CatarinaBelm @sarvk Does this mean that someone with two papers at *ACL conferences and two workshop papers is also qualified?

@VeredShwartz @srchvrs This is incorrect - you can provide non-authors as volunteers as much as you like (something for us to clarify!). [speaking in my role on the tech team helping on this]

@srchvrs I understand the need to give back to the community. I'm just saying it's implemented in a very strict way. For example, you can only nominate a non-author to review if no one else qualifies, which over time may create a weird incentive to add co-authors strategically.

@VeredShwartz @LChoshen @GabiStanovsky Re: that form, responses are merged with other data to produce a spreadsheet for EiCs. For simple cases (e.g., there was no volunteer, now there is), we process it automatically. For unresolved cases, it means that text field is now visible as they decide on desk rejection.

@VeredShwartz @LChoshen @GabiStanovsky ACs are expected to have a load of 4 per submission they are the volunteer for (this was unclear in the form and will be clarified in future cycles)

@yuvalpi @ReviewAcl Any specific suggestions I can pass on for improving communication? ARR does present at conf business meetings, post to its blog, update its website, and has an active account here (though it can't be as responsive as a personal account since it's communication-by-committee)

@yuvalpi @ReviewAcl Yes, this came out before ARR made its announcement, which is a mistake in coordination with the EMNLP PCs, but there definitely will be an announcement

hang on the @ReviewAcl cycles are changing *again*? are you trying to make this system fail? (one would think ongoing harrassment of authors about papers written two years ago would suffice)