Ilia Kuznetsov

33 posts

Ilia Kuznetsov banner
Ilia Kuznetsov

Ilia Kuznetsov

@ilokuznetsov

Postdoc at @UKPLab @TUDarmstadt🌳 // peer review, intertextuality, linguistics, NLP applications // synthesizers 🎹

Darmstadt, Germany Katılım Eylül 2009
137 Takip Edilen72 Takipçiler
Bert Chan
Bert Chan@BertChakovsky·
Galactica: expected science but output bs -> taken down ChatGPT: expected bs and output bs -> big hit
English
15
21
358
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@ani_nenkova Many studies on peer review analysis in NLP and beyond e.g. score calibration, bias, etc. scholar.google.de/scholar?cites=… Also, the problem is charging $60, and not the language quality score per se; nothing bad about language quality scores for peer review texts?
English
0
0
0
0
Ani Nenkova
Ani Nenkova@ani_nenkova·
The idea of computational review analysis makes me uneasy. Especially when part of the motivation is that the regular review process takes a lot of time and is expensive. Help me figure out what will be the useful outcome from this line of investigation, apart from publishing
Ilia Kuznetsov@ilokuznetsov

What is this License Task at @ReviewAcl all about? A peek into ethical and sustainable review data collection, quantitative insights from *ACL, and a place for a public discussion, all in our fresh “Yes-Yes-Yes” preprint here: openreview.net/forum?id=28n-0… @DyNils @IGurevych

English
1
0
3
0
Anna Rogers
Anna Rogers@annargrs·
@ilokuznetsov @ZeerakTalat @ryandcotterell What about retroactive consent retraction? This is a requirement you have to implement under GDPR anyway. People will be more likely to agree to batch contribution if they know they can retracting individual reviews at any point.
English
1
0
1
0
Anna Rogers
Anna Rogers@annargrs·
Some thoughts on the first paper/report on peer review data collection at @ARRPreprints. The basic idea is that if authors+reviewers+editors all agree, data goes to a pubic dataset, otherwise a "protected" dataset only available internally. /1
Ilia Kuznetsov@ilokuznetsov

What is this License Task at @ReviewAcl all about? A peek into ethical and sustainable review data collection, quantitative insights from *ACL, and a place for a public discussion, all in our fresh “Yes-Yes-Yes” preprint here: openreview.net/forum?id=28n-0… @DyNils @IGurevych

English
1
0
3
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@ZeerakTalat @annargrs @ryandcotterell Yes, we were thinking about this; it's just a bit trickier to implement. For now we have NO (default), or yes to all reviews for a *given iteration*. So if for example I don't want to contribute one of my reviews, I just say NO to all of my reviews for this month.
English
1
0
0
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@annargrs @ryandcotterell @ZeerakTalat Good point; retroactive application is definitely not a good idea. We might ask contributors again or leave the data be (or to be used by ACL). Since the process is continuous, the data will get replenished.
English
2
0
0
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@ryandcotterell @annargrs @ZeerakTalat That's what we are discussing now. They way it's often done with protected data is that two institutions (e.g. universities) sign an agreement and then distribute NDA agreements to their members.
English
0
0
0
0
Anna Rogers
Anna Rogers@annargrs·
@ryandcotterell @ilokuznetsov @ZeerakTalat But you still license for specific purposes, right? And they do specify conditions for sublicensing for reviews for the accepted papers, just not for the rejected ones. Yet this whole discussion is about external accessibility of that "internal" data, assuming it's possible.
English
1
0
1
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@annargrs @ryandcotterell @ZeerakTalat Yes by NDA we mean "not sharing the data further and only using it for specific research purposes". It's important to keep track of access to the data; it's both personal data + confidentiality.
English
1
0
0
0
Anna Rogers
Anna Rogers@annargrs·
@ryandcotterell @ilokuznetsov @ZeerakTalat I'm not a lawyer, but the NDA part is meant as a personal data protection mechanism, right? If people are allowed access to internal data for research, they will publish that research -> ACL can't prevent them from disclosing the very fact that the internal data was shared :)
English
1
0
1
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@ryandcotterell @annargrs @ZeerakTalat I do not understand the problem with PEER. 1) As per project page, PEER is annotation based. All we need is preprints. ArXiv has plenty. 2) there is textual data on peer review and papers working on that. It's not licensed and has not been collected in a systematic manner.
English
0
0
0
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@annargrs @ryandcotterell Yes, we're actually writing a blog post now; also to have a TL;DR version of the preprint and have more people involved in the discussion. I'll be posting it here once it's online.
English
0
0
0
0
Anna Rogers
Anna Rogers@annargrs·
@ilokuznetsov @ryandcotterell Ok while that's in progress, maybe there could be a statement clarifying all that, with a timeline estimate + stating that ppl will be able to apply for "protected" data [then] [on conditions], and PEER is in the same boat? Just to avoid the possible perception of a massive COI.
English
1
0
1
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@annargrs @ryandcotterell Yes, absolutely!! But we really need a solution that most people agree on, very clear terms of data sharing, good mechanisms to guarantee protection, and a clear and open option to opt out. That's in progress :)
English
1
0
0
0
Anna Rogers
Anna Rogers@annargrs·
@ryandcotterell @ilokuznetsov Public data is fair game for anyone, of course. I think there'd be a problem only if the "protected" community data was exclusively accessible to one set of researchers from that community. But @ilokuznetsov says that access will be broadened in the coming months?
English
1
0
1
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@ryandcotterell @annargrs Two points need clarification. First, the linked project is not about peer reviewing *automation*, it's about assistance and ML-supported analysis. Whether or not this is useful or the best thing to do is a different point -- guidelines and training are very important.
English
0
0
0
0
Ilia Kuznetsov
Ilia Kuznetsov@ilokuznetsov·
@annargrs Yes, we too! ARR is cool because it has less differences between iterations than regular conferences, so one can really measure the effect of interventions, both qualitatively and quantitatively. ETA ~next months. These things go through ARR EiCs, ACL exec, legal team, etc.
English
0
0
0
0
Anna Rogers
Anna Rogers@annargrs·
@ilokuznetsov Cool. Any ETA on that? I'm not disinterested, admittedly - I've written the reviewer training tutorial, and I really want to know if it made any difference :)
English
1
0
1
0