Robin Shing Moon Chan

PhD Student in HCI and NLP @ ETH ZĂĽrich

robin.chan@inf.ethz.ch

Google Scholar  /  Twitter  /  LinkedIn

About Me

I'm a first-year CS PhD student at the Institute for Visual Computing and the Institute for Machine Learning at ETH ZĂĽrich. I'm co-advised by Prof. Menna El-Assady (IVIA) and Prof. Ryan Cotterell (Rycolab). Previously, I obtained a master's degree in data science from ETH ZĂĽrich, graduating with a thesis on LLM program synthesis at IBM Research Europe (ZRL).

My main research interest lies at the intersection of visualization and natural language processing. More concretely, I want to better understand how to make humans and LLMs interact more effectively. If you're interested in similar topics, message me. I'm looking for collaborators and master's students to supervise.

Personal bits: some of my favorite books: [1, 2, 3], some of my favorite movies: [1, 2, 3]. I like making music [me Christmas caroling with friends].

News
May 27th, 2024 Two papers I co-authored were accepted at the ACL 2024 conference in Bangkok!
March 1st, 2024 Started a PhD at ETH Zürich, co-advised by Prof. Menna El-Assady and Prof. Ryan Cotterell! 🎉
September 26th, 2023 Together with Katya Mirylenka, we will give a talk at Zurich-NLP about our work at IBM Research, at the ETH AI Center. RSVP here!
July 9th, 2023 Our paper on counterfactual sample generation was accepted at ACL. I will be presenting it in Toronto in a few days! Check out our blog post about the paper!
Publications
On Affine Homotopy between Language Encoders
Robin SM Chan, Reda Boumasmoud, Anej Svete, Yuxin Ren, Qipeng Guo, Zhijing Jin, Shauli Ravfogel, Mrinmaya Sachan, Bernhard Schölkopf, Mennatallah El-Assady, Ryan Cotterell
Preprint, 2024 | arXiv
Abstract
Pre-trained language encoders -- functions that represent text as vectors -- are an integral component of many NLP tasks. We tackle a natural question in language encoder analysis: What does it mean for two encoders to be similar? We contend that a faithful measure of similarity needs to be intrinsic, that is, task-independent, yet still be informative of extrinsic similarity -- the performance on downstream tasks. It is common to consider two encoders similar if they are homotopic, i.e., if they can be aligned through some transformation. In this spirit, we study the properties of affine alignment of language encoders and its implications on extrinsic similarity. We find that while affine alignment is fundamentally an asymmetric notion of similarity, it is still informative of extrinsic similarity. We confirm this on datasets of natural language representations. Beyond providing useful bounds on extrinsic similarity, affine intrinsic similarity also allows us to begin uncovering the structure of the space of pre-trained encoders by defining an order over them.

Language Encoders · Metric Spaces · Large Language Models

What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages
Nadav Borenstein, Anej Svete, Robin SM Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, Ryan Cotterell
Published in Proceedings of ACL, 2024. Bangkok, Thailand | arXiv
Abstract
What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formalize it as a matter of learnability of classes of distributions over strings. While prior work in this direction focused on assessing the theoretical limits, in contrast, we seek to understand the empirical learnability. Unlike prior empirical work, we evaluate neural LMs on their home turf—learning probabilistic languages—rather than as classifiers of formal languages. In particular, we investigate the learnability of regular LMs (RLMs) by RNN and Transformer LMs. We empirically test the learnability of RLMs as a function of various complexity parameters of the RLM and the hidden state size of the neural LM. We find that the RLM rank, which corresponds to the size of linear space spanned by the logits of its conditional distributions, and the expected length of sampled strings are strong and significant predictors of learnability for both RNNs and Transformers. Several other predictors also reach significance, but with differing patterns between RNNs and Transformers.

Large Language Models · Formal Language Theory

On Efficiently Representing Regular Languages as RNNs
Anej Svete, Robin SM Chan, Ryan Cotterell
Published in Findings of ACL, 2024. Bangkok, Thailand | arXiv
Abstract
Recent work by Hewitt et al. (2020) provides a possible interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). It shows that RNNs can efficiently represent bounded hierarchical structures that are prevalent in human language. This suggests that RNNs' success might be linked to their ability to model hierarchy. However, a closer inspection of Hewitt et al.'s (2020) construction shows that it is not limited to hierarchical LMs, posing the question of what other classes of LMs can be efficiently represented by RNNs. To this end, we generalize their construction to show that RNNs can efficiently represent a larger class of LMs: Those that can be represented by a pushdown automaton with a bounded stack and a generalized stack update function. This is analogous to an automaton that keeps a memory of a fixed number of symbols and updates the memory with a simple update mechanism. Altogether, the efficiency of representing this diverse class of LMs with RNN LMs suggests novel interpretations of their inductive bias.

Language Model Expressivity · Formal Language Theory

Interactive Analysis of LLMs using Meaningful Counterfactuals
Furui Cheng, Vilém Zouhar, Robin SM Chan, Daniel Fürst, Hendrik Strobelt, Mennatallah El-Assady
Preprint, 2024 | arXiv
Abstract
Counterfactual examples are useful for exploring the decision boundaries of machine learning models and determining feature attributions. How can we apply counterfactual-based methods to analyze and explain LLMs? We identify the following key challenges. First, the generated textual counterfactuals should be meaningful and readable to users and thus can be mentally compared to draw conclusions. Second, to make the solution scalable to long-form text, users should be equipped with tools to create batches of counterfactuals from perturbations at various granularity levels and interactively analyze the results. In this paper, we tackle the above challenges and contribute 1) a novel algorithm for generating batches of complete and meaningful textual counterfactuals by removing and replacing text segments in different granularities, and 2) LLM Analyzer, an interactive visualization tool to help users understand an LLM's behaviors by interactively inspecting and aggregating meaningful counterfactuals. We evaluate the proposed algorithm by the grammatical correctness of its generated counterfactuals using 1,000 samples from medical, legal, finance, education, and news datasets. In our experiments, 97.2% of the counterfactuals are grammatically correct. Through a use case, user studies, and feedback from experts, we demonstrate the usefulness and usability of the proposed interactive visualization tool

Counterfactuals · Visual Explainability

Which Spurious Correlations Impact Reasoning in NLI Models? A Visual Interactive Diagnosis through Data-Constrained Counterfactuals
Robin Chan, Afra Amini, Mennatallah El-Assady
Published in Proceedings of ACL: System Demonstrations, 2023. Toronto, Canada | arXiv | blog post

Abstract
We present a human-in-the-loop dashboard tailored to diagnosing potential spurious features that NLI models rely on for predictions. The dashboard enables users to generate diverse and challenging examples by drawing inspiration from GPT-3 suggestions. Additionally, users can receive feedback from a trained NLI model on how challenging the newly created example is and make refinements based on the feedback. Through our investigation, we discover several categories of spurious correlations that impact the reasoning of NLI models, which we group into three categories: Semantic Relevance, Logical Fallacies, and Bias. Based on our findings, we identify and describe various research opportunities, including diversifying training data and assessing NLI models' robustness by creating adversarial test suites.

Counterfactuals · Mixed-Initiative Learning · Language Model Biases


Website source. Consider using Leonid Keselman's Jekyll fork of this page.