Mengzhou Xia

I’m a fourth-year Computer Science Ph.D. candidate at Princeton NLP, advised by Prof. Danqi Chen. Prior to this, I was a master's student at Carnegie Mellon University, advised by Prof. Graham Neubig. I obtained my Bachelor's degree from Fudan University's School of Data Science in China.

My research is partially supported by the 2024 Apple Scholars in AIML PhD fellowship and the 2022 Bloomberg Data Science Ph.D. Fellowship. I have interned at Meta AI, Microsoft Research, and Bloomberg AI throughout my PhD years.

I am interested in developing safe and capable AI systems through data-centric approaches and innovative objective designs, primarily focusing on large language models.

• Trade-off between computation and model capability: Explore efficient methods for building small-scale yet competitive models through model compression (Sheared Llama, CoFiPruning) and conditional computation (Lory), and study their strengths and limitations through training trajectories.
• Data’s role in shaping model behaviors: Investigate how data influences model capabilities (LESS), safety, and transparency (Benign Data Attack, Mink-Prob), and how data interacts with novel training objectives (SimPO).
• Advancing model reasoning: These days, I am getting particularly excited about further enhancing reasoning capabilities of models (e.g., adapting to novel scenarios efficiently, learning to conduct sequential decision making). We have released several challenging reasoning-intensive benchmarks such as CharXiv and BRIGHT.

[Google Scholar] [Semantic Scholar] [Github] [Twitter] [CV]

News

• [07/2024] I will attend ICLR and ICML in Vienna, and will be co-organizing the High-dimensional Learning Dynamics (HiLD) workshop at ICML!
• [05/2024] Benign data attack won the best paper award at ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models.
• [03/2024] I received the Apple Scholars in AIML PhD fellowship!
• [01/2024] Three papers are accepted to ICLR 2024! See you in Vienna 🥳
• [11/2023] Gave a talk on Sheared-LLaMA at Tencent AI.
• [10/2023] Released the Sheared LLaMA paper on pruning LLMs and the models in 1.3B and 2.7B sizes (downloaded >30K times monthly 🔥)!
• [10/2023] Released a preprint on detecting pre-training data from language models along with UW folks.
• [09/2023] Served as an Area Chair for the Efficient Methods for NLP track for EMNLP 2023.
• [07/2023] Awarded as a Qualcomm Innovation Fellowship finalist.
• [02/2023] Gave a talk at ML Collective on Training Trajectories of LLMs.

Selected Publications and Preprints

For a full list of publications, please refer to this page.

• SimPO: Simple Preference Optimization with a Reference-Free Reward
  Yu Meng*, Mengzhou Xia*, Danqi Chen
  Preprint; [arXiv] [Code]
  


• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
  Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, Danqi Chen
  Preprint; [arXiv] [Code] [Project Page]


• LESS: Selecting Influential Data for Targeted Instruction Tuning
  Mengzhou Xia*, Sadhika Malladi*, Suchin Gururangan, Sanjeev Arora, Danqi Chen
  ICML 2024; [arXiv] [Code] [Blog]


• What's in Your "Safe" Data?: Identifying Benign Data that Breaks Safety
  Luxi He*, Mengzhou Xia*, Peter Henderson
  COLM 2024; DPFM Workshop@ICLR 2024 (Best Paper); [arXiv] [Code]


• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
  Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen
  ICLR 2024; [arXiv] [Code] [Blog]

Awards

• [2024] Apple Scholars in AIML PhD Fellowship.
• [2023] Qualcomm Innovation Fellowship finalist with my labmate Tianyu Gao.
• [2022-2023] Bloomberg Data Science Ph.D. Fellowship.
• [2021-2022] Princeton Graduate Student Teaching Assistant Award.
• [2020-2021] The Hisashi and Masae Kobayashi *67 Fellowship.