Jinlan Fu

Can the text evaluator be customized for different/new evaluation aspects without training? Our GPTScore achieves customized, multifaceted, and training-free using emergent abilities of PLM, i.g., instruction and in-context learning. Paper: arxiv.org/pdf/2302.04166…

Work led by @stefan_fee and me, with @See-Kiong Ng @rucjzb at @NUSingapore and @LTIatCMU Paper: arxiv.org/pdf/2302.04166… Code: github.com/jinlanfu/GPTSc…

[H3] Different evaluation aspects exhibit certain correlations. Combining definitions with other highly correlated aspects can improve evaluation performance.

Can the text evaluator be customized for different/new evaluation aspects without training? Our GPTScore achieves customized, multifaceted, and training-free using emergent abilities of PLM, i.g., instruction and in-context learning. Paper: arxiv.org/pdf/2302.04166…

[1/4] Introducing HyDE, a method to unsupervisedly build dense retrievers. HyDE zero-shot instructs GPT to generate a fictional document and re-encodes it with Contriever to search in its embedding space. Put it simply, casting retrieval-like behavior in GPT into real retrieval.

Q3: What makes a good prompt for multilingual multitask prompt training? A3: The best performance is achieved when the model is trained on the datasets equipped with unified and cross-lingual prompts.

Q2: How do different characteristics of datasets and languages affect the performance of PolyPrompt? A2: PolyPrompt cannot benefit all languages. e.g., languages that appear only once in target datasets have benefits when PolyPrompt is enhanced by high-resource datasets;

Can more conceptually-diverse tasks from different languages communicate together? Our work exploits multilingual prompting methods for learning a unified semantic space for different languages and tasks. #EMNLP2022 Paper: arxiv.org/pdf/2204.14264… Code: github.com/jinlanfu/Polyg…

I'm super honored that our paper was awarded as "Best Demo Paper" at ACL 2021!! Special thanks to all awesome collaborators, reviewers' insightful suggestions, and the committee's recognition. We will continue optimizing ExplainaBoard to make it a truly useful tool!

So excited to share our recent work at #NAACL2021. We empirically study four tagging tasks and thirteen datasets, interpreting the improvement brought by larger-context training using the ExplainaBoard toolkit. explainaboard.nlpedia.ai

XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation We examine the state of multilingual benchmarking and propose an improved benchmark covering more challenging tasks, including a diagnostic and evaluation suite to inform future work. arxiv.org/abs/2104.07412

Happy to be one of the contributors of ExplainaBoard, which covers more than 40 datasets, 9 tasks, and 7 analysis functionalities to interpret the characteristics of systems, datasets, and their interplay. Thanks, @stefan_fee, @gneubig, and et al.. Tool: explainaboard.nlpedia.ai