Home Speakers List Program Day 1 Program Day 2

Day 1 Program (June 3, Mon, Lecture Hall 1 and 2)

9:00 am Opening remarks
Eric Xing (MBZUAI)
9:30 am Self-Play Preference Optimization for Language Model Alignment
Quanquan Gu (UCLA)
Traditional reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this talk, I will introduce a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed \textit{Self-Play Preference Optimization} (SPPO), approximates the Nash equilibrium through iterative policy updates and enjoys theoretical convergence guarantee. Our method can effectively increase the log-likelihood of the chosen response and decrease that of the rejected response, which cannot be trivially achieved by symmetric pairwise loss such as Direct Preference Optimization (DPO) and Identity Preference Optimization (IPO). In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models.
10:00am Beyond Classification: Multimodal Protein Function Prediction with Prot2Text
Hadi Abdine (Ecole Polytechnique)
In recent years, significant advancements have been made in protein function prediction through machine learning. However, most methods limit themselves to multi-classification, assigning predefined labels to proteins. This talk presents Prot2Text, a groundbreaking approach that predicts protein functions in a free text format, transcending traditional binary or categorical classifications. Prot2Text leverages an encoder-decoder framework, combining Graph Neural Networks (GNNs) and Large Language Models (LLMs) to integrate diverse data types such as protein sequences, structures, and textual annotations. This multimodal approach enables a comprehensive representation of protein functions, allowing for the generation of detailed and precise functional descriptions. We will discuss the evaluation of Prot2Text using a multimodal protein dataset extracted from SwissProt and share empirical results demonstrating its effectiveness. This work underscores the transformative potential of integrating GNNs and LLMs, providing researchers with advanced tools for more accurate and nuanced function prediction for both known and novel proteins.
10:30am Coffee Break
11:00am Are large models enough for causality?
Francesco Locatello (ISTA)
Machine learning and AI have the potential to transform data-driven scientific discovery, enabling accurate predictions for several scientific phenomena. Much of the current progress is driven by scale and, conveniently, many scientific questions require analyzing massive amounts of data. At the same time, in scientific applications predictions are often incorporated into broader analysis to draw new insights that are causal in nature. In this talk I will explore the role of scale in causal analyses. I will start by showing it is possible to extract causal knowledge from a pre-trained statistical model that was trained without causal considerations. Next, I will discuss when, how, and why causal directions can be directly predicted by transformers trained with supervised learning in a controlled synthetic setting. Finally, I will discuss the open challenges solving real-world causal downstream tasks in the sciences. For this, I will present ISTAnt, the first real-world benchmark for estimating causal effects from high-dimensional observations in experimental ecology.
11:30am Towards Pixel Grounding Large Language Models
Salman Khan (MBZUAI)
Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this talk, I will talk about Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Further, we propose a densely annotated Grounding-anything Dataset (GranD) using an automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations.
12:00pm Lunch
2:00pm Uncertainty Quantification for Generative Language Models
Maxim Panov (MBZUAI)
The widespread deployment of large language models (LLMs) has made ML-based applications even more vulnerable to risks of causing various forms of harm to users. For example, models often “hallucinate”, i.e., fabricate facts without providing users an apparent means to validate their statements. Uncertainty estimation (UE) methods could be used to detect unreliable generations unlocking the safer and more responsible use of LLMs in practice. UE methods for generative LLMs are a subject of cutting-edge research, which is currently quite scarce and scattered. We systemize these efforts, discuss common caveats, and provide suggestions for the development of novel techniques in this area.
2:30pm Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
Han Zhao (UIUC)
Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, in this talk I will introduce and discuss our Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Fine-tuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).
3:00pm Analyzing the Privacy of Personal Information in Language Models
Nils Hendrik Lukas (MBZUAI)
Language models (LMs) have been shown to leak information about their training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage. Scrubbing reduces but does not prevent the risk of PII leakage. We introduce game-based definitions for three types of PII leakage via black-box extraction, inference, and reconstruction attacks with only API access to the LM. Our main contributions are: (i) novel attacks that can extract up to 10 times more PII sequences than existing attacks, (ii) showing that sentence-level differential privacy reduces the risk of PII disclosure but still leaks about 3% of PII sequences, and (iii) a subtle connection between record-level membership inference and PII reconstruction. I will summarize challenges with privacy-preserving fine-tuning of LMs, describe our proposed solutions, and provide an outlook on challenges and promising future research directions to protect sensitive information in LMs.
3:30pm Coffee Break
4:00pm-5:40pm Special Session: LLM360: DEMYSTIFYING LLM AND TOWARDS OPEN SOURCE AI
Hector Liu (Petuum/MBZUAI)
The rapid advancements in Artificial Intelligence (AI) and Large Language Models (LLMs) have transformed numerous fields, yet their complexities often remain obscure to many. This talk aims to shed light on the building process of LLMs and promote the LLM360 initiative of open-source AI, offering both theoretical insights and practical knowledge for researchers and practitioners alike.

We will discuss the LLM360 project's commitment to open-source principles, highlighting our attempts to democratize AI by making LLM methodologies publicly available. The talk will then delve into the process of pretraining large language models using the LLM360 experiences materials, and provide an overview of research relevant to LLM training. We conclude by presenting a series of case studies that utilize the LLM360 artifacts showcasing LLM behavior and performance.