Nicolas Papernot

I am an Assistant Professor at the University of Toronto, in the Department of Electrical and Computer Engineering and the Department of Computer Science. I am also a faculty member at the Vector Institute where I hold a Canada CIFAR AI Chair, and a faculty affiliate at the Schwartz Reisman Institute. In 2022, I was named an Alfred P. Sloan Research Fellow in Computer Science.

My research interests are at the intersection of security, privacy, and machine learning. If you would like to learn more about my research, I recommend reading the blog posts I co-authored on cleverhans.io, for example about proof-of-learning, collaborative learning beyond federation, dataset inference, machine unlearning, differentially private ML, or adversarial examples.

My research has been cited in the press, including the New York Times, Popular Science, and Wired. I currently serve as a Program Committee Chair of the IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), an Associate Chair of the IEEE Symposium on Security and Privacy (S&P), and an Area Chair of NeurIPS. I earned my Ph.D. in Computer Science and Engineering at the Pennsylvania State University, working with Prof. Patrick McDaniel and supported by a Google PhD Fellowship. Upon graduating, I spent a year at Google Brain where I still spend some of my time.

Email: [email protected]

Office: Pratt 484E and MaRS Suite 710

Mail/Packages: 10 King's College Road, Room SFB540, Toronto, ON M5S 3G4, Canada

CV » Blog » Twitter » Google Scholar »

I am excited to share that Patrick McDaniel and I have been working with IEEE to establish the IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). The SaTML conference will focus on the theoretical and practical understandings of vulnerabilities inherent to ML systems, explore the robustness of ML algorithms and systems, and aid in developing a unified, coherent scientific community which aims to build trustworthy ML systems. More details can be found on our website: satml.org

Recent & selected older publications

A complete list of publications is available in my CV.

2022Washing The Unwashable: On The (Im)possibility of Fairwashing Detection. Ali Shahin Shamsabadi, Mohammad Yaghini, Natalie Dullerud, Sierra Wyllie, Ulrich Aïvodji, Aisha Alaagib Alryeh Mkean, Sébastien Gambs, Nicolas Papernot. Proceedings of the 36th Conference on Neural Information Processing Systems. conferenceDataset Inference for Self-Supervised Models. Adam Dziedzic, Haonan Duan, Muhammad Ahmad Kaleem, Nikita Dhawan, Jonas Guan, Yannis Cattan, Franziska Boenisch, Nicolas Papernot. Proceedings of the 36th Conference on Neural Information Processing Systems. conferenceIn Differential Privacy, There is Truth: on Vote-Histogram Leakage in Ensemble Private Learning. Jiaqi Wang, Roei Schuster, Ilia Shumailov, David Lie, Nicolas Papernot. Proceedings of the 36th Conference on Neural Information Processing Systems. conferenceOn the Limitations of Stochastic Pre-processing Defenses. Yue Gao, Ilia Shumailov, Kassem Fawaz, Nicolas Papernot. Proceedings of the 36th Conference on Neural Information Processing Systems. conferenceOn the Fundamental Limits of Formally (Dis)Proving Robustness in Proof-of-Learning. Congyu Fang, Hengrui Jia, Anvith Thudi, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Varun Chandrasekaran, Nicolas Papernot. preprintSelective Classification Via Neural Network Training Dynamics. Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, Nicolas Papernot. preprintOn the Difficulty of Defending Self-Supervised Learning against Model Extraction. Adam Dziedzic, Nikita Dhawan, Muhammad Ahmad Kaleem, Jonas Guan, Nicolas Papernot. Proceedings of the 39th International Conference on Machine Learning. conferenceUnrolling SGD: Understanding Factors Influencing Machine Unlearning. Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, Nicolas Papernot. Proceedings of the 7th IEEE European Symposium on Security and Privacy, Genoa, Italy. conferenceOn the Necessity of Auditable Algorithmic Definitions for Machine Unlearning. Anvith Thudi, Hengrui Jia, Ilia Shumailov, Nicolas Papernot. Proceedings of the 31st USENIX Security Symposium. conferenceIncreasing the Cost of Model Extraction with Calibrated Proof of Work. Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, Nicolas Papernot. Proceedings of the 10th International Conference on Learning Representations. conference (+spotlight)A Zest of LIME: Towards Architecture-Independent Model Distances. Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, Nicolas Papernot. Proceedings of the 10th International Conference on Learning Representations. conferenceHyperparameter Tuning with Renyi Differential Privacy. Nicolas Papernot, Thomas Steinke. Proceedings of the 10th International Conference on Learning Representations. conference (+outstanding paper award)Is Fairness Only Metric Deep? Evaluating and Addressing Subgroup Gaps in Deep Metric Learning. Natalie Dullerud, Karsten Roth, Kimia Hamidieh, Nicolas Papernot, Marzyeh Ghassemi. Proceedings of the 10th International Conference on Learning Representations. conferenceBad Character Injection: Imperceptible Attacks on NLP Models. Nicholas Boucher, Ilia Shumailov, Ross Anderson, Nicolas Papernot. Proceedings of the 43rd IEEE Symposium on Security and Privacy, San Francisco, CA. conferenceTowards More Robust Keyword Spotting for Voice Assistants. Shimaa Ahmed, Ilia Shumailov, Nicolas Papernot, Kassem Fawaz. Proceedings of the 31st USENIX Security Symposium. conference2021When the Curious Abandon Honesty: Federated Learning Is Not Private. Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, Nicolas Papernot. preprintManipulating SGD with Data Ordering Attacks. Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A. Erdogdu, Ross Anderson. Proceedings of the 35th Conference on Neural Information Processing Systems. conferenceData-Free Model Extraction. Jean-Baptiste Truong, Pratyush Maini, Robert Walls, Nicolas Papernot. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN. conferenceProof-of-Learning: Definitions and Practice. Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, Nicolas Papernot. Proceedings of the 42nd IEEE Symposium on Security and Privacy, San Francisco, CA. conferenceEntangled Watermarks as a Defense against Model Extraction. Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, Nicolas Papernot. Proceedings of the 30th USENIX Security Symposium. conferenceSponge Examples: Energy-Latency Attacks on Neural Networks. Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, Ross Anderson. Proceedings of the 6th IEEE European Symposium on Security and Privacy, Vienna, Austria. conferenceCaPC Learning: Confidential and Private Collaborative Learning. Christopher A. Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, Xiao Wang. Proceedings of the 9th International Conference on Learning Representations. conferenceDataset Inference: Ownership Resolution in Machine Learning. Pratyush Maini, Mohammad Yaghini, Nicolas Papernot. Proceedings of the 9th International Conference on Learning Representations. conference (+spotlight)Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings. Vinith Suriyakumar, Nicolas Papernot, Anna Goldenberg, Marzyeh Ghassemi. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. conferenceMachine Unlearning. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot. Proceedings of the 42nd IEEE Symposium on Security and Privacy, San Francisco, CA. conference2020 & earlierAnalyzing and Improving Representations with the Soft Nearest Neighbor Loss. Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA. conferenceDeep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. Nicolas Papernot and Patrick McDaniel. technical reportScalable Private Learning with PATE. Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, Ulfar Erlingsson. Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada. conferenceTowards the Science of Security and Privacy in Machine Learning. Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael Wellman. Proceedings of the 3rd IEEE European Symposium on Security and Privacy, London, UK. conferenceSemi-supervised Knowledge Transfer for Deep Learning from Private Training Data. Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Proceedings of the 5th International Conference on Learning Representations, Toulon, France. conference (+best paper)Practical Black-Box Attacks against Machine Learning. Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z.Berkay Celik, and Ananthram Swami. Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security, Abu Dhabi, UAE. conferenceTransferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. technical reportThe Limitations of Deep Learning in Adversarial Settings. Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. Proceedings of the 1st IEEE European Symposium on Security and Privacy, Saarbrucken, Germany. conference

Research group

Current students and postdocs
Haonan Duan: PhD student (started Fall 2021, co-advised with Chris Maddison)Camille Bruckmann: Engineering Science student (Fall 2022 - Summer 2023)Si Cheng (Steven) Zhong: Engineering Science student (Fall 2022 - Summer 2023)Franziska Boenisch: Postdoctoral Fellow (started Fall 2022)David Glukhov: MS student (started Fall 2022, co-advised with Vardan Papyan) OGS ScholarAnvith Thudi: PhD student (started Fall 2022, co-advised with Chris Maddison)Patty Liu: Research Intern (Started May 2022)Roei Schuster: Postdoctoral Fellow (started Fall 2021)Ilia Shumailov: Postdoctoral Fellow (started Fall 2021, co-advised with Kassem Fawaz)Aditi Misra: Engineering Science student (started Fall 2021)Sierra Wyllie: Engineering Science student (started Summer 2021)Muhammad Ahmad Kaleem: Engineering Science student (started Summer 2021)Emmy Fang: MS student (started Fall 2021, co-advised with Bo Wang) DeepMind ScholarAdam Dziedzic: Postdoctoral Fellow (started Fall 2020)Mohammad Yaghini: PhD student (started Fall 2020) Meta PhD FellowStephan Rabanser: PhD student (started Fall 2020)Jonas Guan: PhD student (started Fall 2020)Jiaqi Wang: MASc student (started Fall 2020, co-advised with David Lie) OGS ScholarNick Jia: PhD student (started Fall 2020) Mary H. Beatty FellowMingyue Yang: PhD student (started Winter 2020, co-advised with David Lie)
Past students and postdocs
Shimaa Ahmed: Research Intern (Summer 2022) currently PhD student at University of Wisconsin-MadisonRoy Rinberg: Research Intern (Summer 2022) currently Masters student at Columbia UniversityMark Thomas: Research Intern (Summer 2022) currently Honors Computing Science student at the University of AlberaAvital Shafran: Research Intern (Summer 2022) currently PhD student at the Hebrew University of JerusalemThorsten Eisenhofer: Research Intern (Summer 2022) currently PhD student at Ruhr University BochumYannis Cattan: Research Intern (Summer 2022) currently Masters student at ENS Paris-Saclay (MVA)Hongyu (Charlie) Chen: Engineering Science student (Fall 2021 - Summer 2022) currently Machine Learning Engineer at Cohere.aiAisha Alaagib: Research Intern (Summer 2021) currently PhD student at MILAArmin Ale: Engineering Science student (Summer 2021 - Summer 2022) currently Software Engineer at IntelAli Shahin Shamsabadi: Research Intern (Winter 2021 - Fall 2021) currently Research Associate at the Turing InstituteNatalie Dullerud: MS student (Fall 2020 - Summer 2022) currently PhD Student at StanfordSteven Xia: Undergraduate student (Fall 2020 - Summer 2021, co-advised with Shurui Zhou) currently PhD student at UIUCJin Zhou: Engineering Science student (Fall 2020 - Summer 2021) currently PhD student at CornellLucy Lu: Engineering Science student (Fall 2020 - Summer 2021) currently MS student at StanfordMarko Huang: Engineering Science student (Fall 2020 - Summer 2021) currently MS student at University of TorontoGabriel Deza: Engineering Science student (Fall 2020 - Summer 2021) currently MS student at UC BerkeleyTejumade Afonja: Research Intern (Summer 2020) currently MS student at Saarland UniversityMilad Nasr: Google Brain Intern (Summer 2020, co-hosted with Nicholas Carlini) currently Research Scientist at Google BrainLorna Licollari: Research Intern (Summer 2020) currently Engineering Science student at University of TorontoPratyush Maini: Research Intern (Summer 2020) currently PhD student at CMUYunxiang Zhang: Research Intern (Spring 2020) currently PhD student at Chinese University of Hong KongSaina Asani: Research Assistant (Winter 2020 - Summer 2020) currently AI Researcher at HuaweiLaura Zhukas: Undergraduate Student Researcher (Fall 2019) currently BASc student at the University of WaterlooChristopher Choquette-Choo: Engineering Science student (Fall 2019 - Summer 2020) currently Research Engineer at Google BrainBaiwu Zhang: MEng student (Fall 2019 - Summer 2020) currently ML Engineer at TwitterVarun Chandrasekaran: Visiting PhD student (Fall 2019) currently Assistant Professor at UIUCVinith Suriyakumar: MS student (Fall 2019 - Summer 2021, co-advised with M. Ghassemi and A. Goldenberg) currently PhD student at MITLucas Bourtoule: MASc student (started Fall 2019) currently Cybersecurity Software Engineer at Mithril SecurityAdelin Travers: PhD student (Fall 2019 - Summer 2021, co-advised with David Lie) currently Senior Pentester at VerizonHadi Abdullah: Google Intern (Summer 2019, co-hosted with Damien Octeau) currently Researcher at Visa ResearchMatthew Jagielski: Google Brain intern (Summer 2019) currently Research Scientist at Google Brain
Information for prospective graduate students and postdocs
If you are interested in joining my research group as a graduate student, apply to the CS or ECE (select "software systems" field) program. Unfortunately, I cannot respond to all prospective graduate students, so the best time is to contact me after you submitted your application. If you are interested in joining my research group as a postdoc, please send me an email directly with your CV and research statement.

Research Talks

Upcoming

Here is a list of talks I will be giving. Feel free to reach out if you will be attending one of these events and would like to meet.

11/2022 - ACM CCS Workshop on Moving Target Defense keynote10/2022 - University of Pittsburgh lecture10/2022 - University of Waterloo panel10/2022 - CISPA9/2022 - Georgetown University
Past Recorded Talks

These video resources are a good overview of my research interests.

Tempered Sigmoids for DP-SGD Trustworthy ML Deepfakes Lecture on ML security and privacy Privacy-preserving ML Adversarial examples

Blog Posts

Here is a list of blog posts discussing some of the research questions I'm interested in:

Are adversarial examples against proof-of-learning adversarial?How to Keep a Model Stealing Adversary Busy?All You Need Is MatplotlibHow to deploy machine learning with differential privacy? (DifferentialPrivacy.org)Arbitrating the integrity of stochastic gradient descent with proof-of-learningBeyond federation: collaborating in ML with confidentiality and privacyIs this model mine?Why we should regulate information about persons, not personal informationTo guarantee privacy, focus on the algorithms, not the dataTeaching Machines to UnlearnIn Model Extraction, Don’t Just Ask How?: Ask Why?How to steal modern NLP systems with gibberish?The academic job search for computer scientists in 10 questionsHow to know when machine learning does not knowMachine Learning with Differential Privacy in TensorFlowPrivacy and machine learning: two unexpected allies?The challenge of verification and testing of machine learningIs attacking machine learning easier than defending it?Breaking things is easy

Teaching

[Fall 2022] ECE1784H/CSC2559H: Trustworthy Machine Learning[Fall 2022] ECE421H: Introduction to Machine Learning (see Quercus for course details)[Fall 2021] ECE1784H/CSC2559H: Trustworthy Machine Learning[Fall 2021] ECE421H: Introduction to Machine Learning (see Quercus for course details)[Fall 2020] ECE421H: Introduction to Machine Learning (see Quercus for course details)[Fall 2020] ECE1513H: Introduction to Machine Learning (see Quercus for course details)[Winter 2020] ECE1513H: Introduction to Machine Learning (see Quercus for course details)[Fall 2019] ECE1784H: Trustworthy Machine Learning