Logo of the University of Notre Dame

About me

Toby Jia-Jun Li is an Assistant Professor in the Department of Computer Science and Engineering at the University of Notre Dame where he directs the SaNDwich Lab. Toby received a Ph.D. degree in Human-Computer Interaction at Carnegie Mellon University, where he was advised by Brad A. Myers. Toby also worked closely with Tom M. Mitchell. Prior to Ph.D., Toby received his Bachelor’s degree in Computer Science from the University of Minnesota where he worked with Brent Hecht as a member of GroupLens Research.

Toby works at the intersection of Human-Computer Interaction (HCI), End-User Software Engineering, Machine Learning (ML), and Natural Language Processing (NLP) applications, where he uses human-centered methods to design, build, and study interactive systems to empower individuals to create, configure, and extend AI-powered computing systems. His recent work seeks to address the societal challenges in the future of work through a bottom-up human-AI collaborative approach that helps individual workers automate and augment their tasks with AI systems.

Toby publishes at premier academic venues across HCI, NLP, and systems (e.g., CHI, UIST, CSCW, ACL, MobiSys, VL/HCC), including 4 award-winning papers: a best paper honorable mention paper at CHI ‘21, a best paper at UIST ‘20, a best paper at IS-EUD ‘17, and a best paper honorable mention paper at CHI ‘17. Toby has collaborated with Microsoft Research, IBM Research, Verizon, and J.P. Morgan. His work has been supported by NSF, the Google Research Scholar Program, the AnalytiXIN Initiative, Yahoo! through the InMind project, and J.P. Morgan.

📢 I am recruiting Ph.D. students, undergraduate researchers, and visiting researchers. See the Students page for details.

News

2022-06-21
💰 Happy to share that we, in collaboration with the groups of Collin McMillan at Notre Dame and Yu Huang at Vanderbilt University, have received a 4-year NSF grant to study human-like AI models of source code that power AI-enabled developer tools for e.g., code summarization, code completion.
2022-06-01
🏆 Honored to receive a Google Research Scholar Award to support our work on human-AI collab in UX design! Thanks Google Research for the generous support!
2022-05-22
📢 Check out our ACL 2022 papers! We have two papers in collaboraiton with researchers at UC Irvine, RPI, IBM Research, and other institutions on new NLP techinques and datasets for assisting and augmenting children’s interactive storyreading.
2022-04-29
🥪 SaNDwich Lab is at CHI 2022! Check out Zheng Zhang’s presentation of our paper StoryBuddy: A Human-AI Collaborative Agent for Parent-Child Interactive Storytelling with Flexible Parent Involvement, Yuwen Lu’s poster presentation on Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools, and the workshop we organize on Computational Approaches for Understanding, Generating, and Adapting User Interfaces!
2021-12-16
📢 We are organizing a workshop on Computational Approaches for Understanding, Generating, and Adapting User Interfaces at CHI 2022!
2021-10-28
🏅 My Ph.D. disseration “A Multi-Modal Intelligent Agent that Learns from Demonstrations and Natural Language Instructions” won an honorable mention for CMU School of Computer Science Dissertation Award!
2021-10-16
📃 My Ph.D. student Zheng Zhang presented our paper “Building an Interactive Storytelling Conversational Agent through Parent-AI Collaboration” at CSCW 2021 Workshop on Inclusive and Collaborative Child-facing Voice Technologies (CUI@CSCW).
2021-03-14
🏅 Our paper “Screen2Vec: Semantic Embedding of GUI Screens and GUI Components” won a Best Paper Honorable Mention Award at CHI 2021!
2020-12-19
🏆 My wonderful summer REU student Lindsay Popowski won CRA’s 2021 Outstanding Undergraduate Researcher Award!
2020-12-12
📃 Our paper “Screen2Vec: Semantic Embedding of GUI Screens and GUI Components” has been accepted to CHI 2021! The code is also open-sourced here.
2020-08-28
🏆 Our paper “Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs” won the Best Paper Award at UIST 2020!
2020-07-21
📃 Two papers accepted at UIST 2020! The paper “Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs” introduces a new multi-modal interface for users to discover and repair conversational breakdowns in task-oriented dialogs. The other paper “Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications” is a collaboration with Xiang ‘Anthony’ Chen’s group at UCLA on a new developer tool for authoring multi-modal interfaces for web applications!
2020-04-07
📃 Our paper “Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems” has been accepted to the CSCW 2020! This work is done with my wonderful interns Jingya Chen and Brandon Canfield.
2020-02-27
📃 Our position paper “Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents” has been accepted to the CHI 2020 Workshop on Artificial Intelligence for HCI: A Modern Approach!
2019-12-13
💬 I’m giving an invited talk at The AAAI-20 Workshop on Intelligent Process Automation (IPA 20). I hope you can come if you’ll be at AAAI 2020!

MORE...

2019-11-26
📃 My thesis proposal document is available online, and the video of my talk is available too!
2019-08-01
📃 Our paper “PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations” has been accepted to UIST 2019!
2018-10-02
💬 My awesome summer intern Marissa Radensky will present our study How End Users Express Conditionals in Programming by Demonstration for Mobile Apps at VL/HCC 2018!
2018-07-31
📃 Our paper “APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions” has been accepted to VL/HCC 2018!
2018-04-22
💬 I will talk about our position paper “Supporting Co-adaptive Human-Agent Relationship through Programming by Demonstration using Existing GUIs” at CHI 2018 Workshop on Rethinking Interaction.
2018-02-28
📃 Our paper “KITE: Building conversational bots from mobile apps” has been accepted to MobiSys 2018!
2017-06-15
🏆 We have received a Best Paper award at IS-EUD 2017 for our paper “Programming IoT Devices by Demonstration Using Mobile Apps”. https://www.nd.edu/* 2017-03-22
📃 Our paper “Programming IoT Devices by Demonstration Using Mobile Apps” has been accepted to The 6th International Symposium on End User Development (IS-EUD 2017).
2017-02-24
💬 I will present our work on Designing a Conversational Interface for a Multimodal Smartphone Programming-by-Demonstration Agent at the Conversational UX Design CHI 2017 Workshop.
2017-02-17
🏅 We have received a Best Paper Honorable Mention award at CHI 2017 for our paper “SUGILITE: Creating Multimodal Smartphone Automation by Demonstration”.
2017-01-06
📃 Our paper “SUGILITE: Creating Multimodal Smartphone Automation by Demonstration” has been accepted to CHI ’17! A Video DEMO is also available.
2016-03-10
🏅 I’m honored to receive the Yahoo! Fellowship ($100,000/year worth) to support my research in automating cross-application tasks on smartphone.
2016-02-16
💬 I will present our ongoing work “Smartphone Text Entry in Cross-Application Tasks” on CHI 2016 Workshop on Inviscid Text Entry and Beyond at San Jose, CA on May 8, 2016
2015-04-15
🏫 Happy to announce that I’ll be joining HCII @ CMU as a PhD student in HCI starting next fall!

Travel

2022-06-08 – 2022-06-09
Durham, NH for CHIWORK 2022
2022-04-30 – 2022-05-06
New Orleans, LA for CHI 2022
2022-02-23 – 2022-02-25
Washington, DC for CRA Career Mentoring Workshop
2021-11-08 – 2020-11-10
Pittsburgh, PA for PLATEAU 2021
2021-05-08 – 2021-05-13
~~Yokohama, Japan for CHI 2021~~ (Cancelled due to COVID-19)
2020-10-17 – 2020-10-23
~~Minneapolis, MN for CSCW 2020 and UIST 2020~~ (Cancelled due to COVID-19)
2020-07-05 – 2020-07-10
~~Seattle, WA for ACL 2020~~ (Cancelled due to COVID-19)
2020-04-25 – 2020-04-30
~~Honolulu, HI for CHI 2020~~ (Cancelled due to COVID-19)
2020-02-06 – 2020-02-09
New York, NY for AAAI 2020

MORE...

2019-10-20 – 2019-10-23
New Orleans, LA for UIST 2019
2019-05-23 – 2019-05-24
New York, NY for giving a talk at J.P. Morgan
2019-05-04 – 2019-05-09
Glasgow, Scotland for CHI 2019
2019-03-17 – 2019-03-20
Los Angeles, CA for IUI 2019
2018-10-01 – 2018-10-04
Lisbon, Portugal for VL/HCC 2018
2018-06-24 – 2018-06-28
Pajaro Dunes, Watsonville, CA for HCIC 2018
2018-06-10 – 2018-06-15
Munich, Germany for MobiSys 2018
2018-05-29 – 2018-05-31
Sunnyvale, CA for giving a talk at Oath
2018-04-21 – 2018-04-26
Montreal, QC for CHI 2018
2018-01-08 – 2018-01-15
Ithaca, NY
2017-10-11 – 2017-10-14
Raleigh, NC for VL/HCC 2017
2017-06-13 – 2017-06-15
Eindhoven, The Netherlands for IS-EUD 2017
2017-05-22 – 2017-08-18
Redmond, WA – Intern at Microsoft Research
2017-05-06 – 2017-05-12
Denver, CO for CHI 2017
2016-12-30 – 2017-01-01
Ottawa, ON
2016-12-28 – 2016-12-30
Toronto, ON
2016-05-07 – 2016-05-12
San Jose, CA for CHI 2016
2014-11-04 – 2014-11-07
Dallas, TX for SIGSPATIAL 2014

Recent Publications

CHIWORK 2022

A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality
Toby Jia-Jun Li, Yuwen Lu, Jaylexia Clark, Meng Chen, Victor Cox, Meng Jiang, Yang Yang, Tamara Kay, Danielle Wood, and Jay Brockman
Proceedings of the 1st Symposium on Human-Computer Interaction for Work (CHIWORK 2022)
[Paper PDF] [ACM DL]

ACL 2022

It is AI’s Turn to Ask Human a Question: Question and Answer Pair Generation for Children Storybooks in FairytaleQA Dataset
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Mo Yu, and Ying Xu
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)
[Paper PDF] [ACL Anthology]

ACL 2022

Fantastic Questions and Where to Find Them: FairytaleQA–An Authentic Dataset for Narrative Comprehension
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)
[Paper PDF] [ACL Anthology]

CHI 2022

StoryBuddy: A Human-AI Collaborative Agent for Parent-Child Interactive Storytelling with Flexible Parent Involvement
Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI 2022)
[Paper PDF] [ACM DL]

CHI 2021

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [GitHub]

UIST 2020

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, and Brad A. Myers
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
🏆 Best Paper Award
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

UIST 2020

Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
Ritam Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang ‘Anthony’ Chen
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
[Paper PDF] [ACM DL] [Demo Video]

CSCW 2020

Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, and Brad A. Myers
Proceedings of the ACM on Human-Computer Interaction (CSCW 2020)
[Paper PDF] [ACM DL]

ACL 2020 Demo

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
Toby Jia-Jun Li, Tom Mitchell, and Brad A. Myers
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020): System Demonstrations
[Paper PDF] [ACL Anthology] [Talk Video]

UIST 2019

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST 2019)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

VL/HCC 2018

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018)
[Paper PDF] [IEEE Xplore] [Demo Video]

MobiSys 2018

KITE: Building conversational bots from mobile apps
Toby Jia-Jun Li and Oriana Riva
Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018)
[Paper PDF] [ACM DL] [Talk Video]

IMWUT/Ubicomp 2017

PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps
Yuanchun Li, Fanglin Chen, Toby Jia-jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, and Jason I. Hong
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2017)
[Paper PDF] [ACM DL] [Website]

IS-EUD 2017

Programming IoT Devices by Demonstration Using Mobile Apps
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A. Myers
Proceedings of the International Symposium on End User Development (IS-EUD 2017)
🏆 Best Paper Award
[Paper PDF] [SpringerLink]

CHI 2017

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

CHI 2016

Not at Home on the Range: Peer Production and the Urban/Rural Divide
Isaac Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016)
[Paper PDF] [ACM DL]

SIGSPATIAL 2014

Leveraging Advances in Natural Language Processing to Better Understand Tobler’s First Law of Geography
Toby Jia-Jun Li, Shilad Sen, and Brent Hecht
Proceedings of the ACM Conference on Advances in Geographic Information Systems (SIGSPATIAL 2014)
[Paper PDF] [ACM DL]

OpenSym / WikiSym 2014

WikiBrain: Democratizing Computation on Wikipedia
Shilad Sen, Toby Jia-Jun Li, WikiBrain Team, and Brent Hecht
Proceedings of the 10th International Symposium on Open Collaboration (OpenSym / WikiSym 2014)
[Paper PDF] [ACM DL]