Noam Razin (Princeton University) – Fundamentals of Aligning General-Purpose AI

Abstract:
The field of artificial intelligence (AI) is undergoing a paradigm shift, moving from neural
networks trained for narrowly defined tasks (e.g., image classification and machine translation)
to general-purpose models such as ChatGPT. These models are trained at unprecedented
scales to perform a wide range of tasks, from providing travel recommendations to solving
Olympiad-level math problems. As they are increasingly adopted in society, a central challenge
is to ensure the alignment of general-purpose models with human preferences. In this talk, I will
present a series of works that reveal fundamental pitfalls in existing alignment methods. In
particular, I will show that they can: (1) suffer from a flat objective landscape that hinders
optimization, and (2) fail to reliably increase the likelihood of generating preferred outputs,
sometimes even causing the model to generate outputs with an opposite meaning. Beyond
characterizing these pitfalls, our theory provides quantitative measures for identifying when they
occur, suggests preventative guidelines, and has led to the development of new data selection
and alignment algorithms, validated at large scale in real-world settings. Our contributions
address both efficiency challenges and safety risks that may arise in the alignment process. I
will conclude with an outlook on future directions, toward building a practical theory in the age of
general-purpose AI.

Short bio:
Noam Razin is a Postdoctoral Fellow at Princeton Language and Intelligence, Princeton
University. His research focuses on the fundamentals of artificial intelligence (AI). By combining mathematical analyses with systematic experimentation, he aims to develop theories that shed light on how modern AI works, identify potential failures, and yield principled methods for improving efficiency, reliability, and performance.
Noam earned his PhD in Computer Science at Tel Aviv University, where he was advised by Nadav Cohen. Prior to that, he obtained a BSc in Computer Science (summa cum laude) at The
Hebrew University of Jerusalem under the Amirim honors program. For his research, Noam received several honors and awards, including the Zuckerman Postdoctoral Scholarship, the Israeli Council for Higher Education (VATAT) Postdoctoral Scholarship, the Apple Scholars in AI/ML PhD fellowship, the Tel Aviv University Center for AI and Data Science excellence fellowship, and the Deutsch Prize for PhD candidates.