Logo of the University of Notre Dame

About me

Toby Jia-Jun Li is an Assistant Professor in the Department of Computer Science and Engineering at the University of Notre Dame, where he leads the SaNDwich Lab. He also serves as the Director of the Human-Centered Responsible AI Lab in the Lucy Family Institute for Data & Society and a Faculty Fellow at the Institute for Educational Initiatives (IEI). Toby received his Ph.D. in Human-Computer Interaction at Carnegie Mellon University, where he was advised by Brad A. Myers. Toby also worked closely with Tom M. Mitchell. He received his Bachelor’s degree in Computer Science from the University of Minnesota, where he worked with Brent Hecht in GroupLens Research.

Toby works at the intersection of Human-Computer Interaction (HCI), End-User Software Engineering, Machine Learning (ML), and Natural Language Processing (NLP) applications, where he uses human-centered methods to design, build, and study interactive systems to empower individuals to create, configure, and extend AI-powered computing systems. His recent work seeks to address the societal challenges in the future of work through a bottom-up human-AI collaborative approach that helps individual workers automate and augment their tasks with AI systems.

Toby publishes at premier academic venues across HCI, NLP, and systems (e.g., CHI, UIST, CSCW, ACL, EMNLP, MobiCom, MobiSys, ICSE, FSE, VL/HCC), including 8 papers that won Best Paper or Best Paper Honorable Mention awards. Toby has collaborated with Google, Microsoft Research, IBM Research, Adobe, Verizon, and J.P. Morgan. His work has been supported by NSF, the Google Research Scholar Program, the AnalytiXIN Initiative, Yahoo! through the InMind project, and J.P. Morgan.

📢 I am recruiting Ph.D. students, undergraduate researchers, and visiting researchers. See the Students page for details.

News

2024-11-04
🏆 The paper “From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX”, led by our Ph.D. student Yuwen Lu in collaboration with Chao Zhang, Yuewen Yang, and Yaxing Yao won a Best Paper Award at CSCW 2024!
2024-08-12
💰 Our team (with Jill Pentimonti, Rick Johnson, and Tricia Zucker) has received an EAGER grant from NSF to develop AI-powered targeted instructional support for early childhood teachers.
2024-07-15
💰 Our team (with Yaxing Yao from Virginia Tech, Tianshi Li from Northeastern, and Julia Qian from ND) has received a SaTC CORE grant from NSF to develop new LLM-based technologies for empathy-based privacy education and design through synthetic persona data generation.
2024-07-01
💰 Our highly interdisciplinary team across Computer Science, Writing (Nathaniel Myers and Patrick Clauss), English (Ranjodh Singh Dhaliwal), and Education (Alex Ambrose) will design, deploy, and study the thoughtful and ethical integration of generative AI in undergraduate writing education with the support of a Strategic Framework Teaching Grant by Notre Dame’s Provost Office as a part of the new Notre Dame Strategic Framework!
2024-05-14
💰 Excited to receive funding to work with folks at IBM Research on the evaluation, metrics, and benchmarks for generative AI systems through the the Notre Dame–IBM Technology Ethics Lab, together with Diego Gómez-Zará!
2024-05-11
🥪 SaNDwich Lab at CHI 2024! The Notre Dame HCI group has 10 full papers accepted by CHI 2024, including 4 co-authored by members of the SaNDwich Lab. Learn more about our CHI 2024 papers here. We also have several other case studies, late breaking work papers, and workshops at CHI 2024.
2024-04-22
💰 Our teams with collaborators at CMU and Northeastern University have received two awards from the the Notre Dame–IBM Technology Ethics Lab.
2023-09-13
💰 Glad to share that our interdisciplinary team across Computer Science, Computational Social Science, Business, and Labor Sociology (Co-PIs: Meng Jiang, Tamara Kay, Yang Yang, Jay Brockman; Senior Personnel: Danielle Wood) has received a 4-year $1 million NSF grant to develop a new bottom-up method aiming to bridge AI inequality in digitally mediated gig work.
2023-08-21
🥪 SaNDwich Lab at UIST 2023! The paper “VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping”, led by Zheng Zhang, introduces a new AI-enabled writing assistant system for argumentative writing. The other paper “PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data”, also led by Zheng Zhang, presents a data annotation tool that significantly accelerates the audio-visual data annotation process while maintaining high annotation accuracy.
2023-01-17
📃 Happy to share that my Ph.D. student Simret Gebreegziabher will present a paper PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis at CHI 2023! Congrats to Simret for her first CHI paper. My PhD student Yuwen Lu and I are also organizers (along with several other leaders in this filed) of the 2nd iterations of the Computational UI Workshop at CHI 2023.
2023-01-03
📃 If you are going to IUI 2023, make sure to check out Zheng Ning’s presentation of our paper An Empirical Study of Model Errors and User Error Discovery and Repair Strategies in Natural Language Database Queries! I also look forward to seeing you at the SHAI 2023 Workshop on Designing for Safety in Human-AI Interactions.
2022-06-21
💰 Happy to share that we, in collaboration with the groups of Collin McMillan at Notre Dame and Yu Huang at Vanderbilt University, have received a 4-year $1.3 million NSF grant to study human-like AI models of source code that power AI-enabled developer tools for e.g., code summarization, code completion.
2022-06-01
🏆 Honored to receive a Google Research Scholar Award to support our work on human-AI collab in UX design! Thanks Google Research for the generous support!
2022-05-22
📢 Check out our ACL 2022 papers! We have two papers in collaboraiton with researchers at UC Irvine, RPI, IBM Research, and other institutions on new NLP techinques and datasets for assisting and augmenting children’s interactive storyreading.
2022-04-29
🥪 SaNDwich Lab is at CHI 2022! Check out Zheng Zhang’s presentation of our paper StoryBuddy: A Human-AI Collaborative Agent for Parent-Child Interactive Storytelling with Flexible Parent Involvement, Yuwen Lu’s poster presentation on Bridging the Gap Between UX Practitioners’ Work Practices and AI-Enabled Design Support Tools, and the workshop we organize on Computational Approaches for Understanding, Generating, and Adapting User Interfaces!
2021-12-16
📢 We are organizing a workshop on Computational Approaches for Understanding, Generating, and Adapting User Interfaces at CHI 2022!
2021-10-28
🏅 My Ph.D. disseration “A Multi-Modal Intelligent Agent that Learns from Demonstrations and Natural Language Instructions” won an honorable mention for CMU School of Computer Science Dissertation Award!
2021-10-16
📃 My Ph.D. student Zheng Zhang presented our paper “Building an Interactive Storytelling Conversational Agent through Parent-AI Collaboration” at CSCW 2021 Workshop on Inclusive and Collaborative Child-facing Voice Technologies (CUI@CSCW).
2021-03-14
🏅 Our paper “Screen2Vec: Semantic Embedding of GUI Screens and GUI Components” won a Best Paper Honorable Mention Award at CHI 2021!
2020-12-19
🏆 My wonderful summer REU student Lindsay Popowski won CRA’s 2021 Outstanding Undergraduate Researcher Award!
2020-12-12
📃 Our paper “Screen2Vec: Semantic Embedding of GUI Screens and GUI Components” has been accepted to CHI 2021! The code is also open-sourced here.
2020-08-28
🏆 Our paper “Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs” won the Best Paper Award at UIST 2020!

MORE...

2020-07-21
📃 Two papers accepted at UIST 2020! The paper “Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs” introduces a new multi-modal interface for users to discover and repair conversational breakdowns in task-oriented dialogs. The other paper “Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications” is a collaboration with Xiang ‘Anthony’ Chen’s group at UCLA on a new developer tool for authoring multi-modal interfaces for web applications!
2020-04-07
📃 Our paper “Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems” has been accepted to the CSCW 2020! This work is done with my wonderful interns Jingya Chen and Brandon Canfield.
2020-02-27
📃 Our position paper “Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents” has been accepted to the CHI 2020 Workshop on Artificial Intelligence for HCI: A Modern Approach!
2019-12-13
💬 I’m giving an invited talk at The AAAI-20 Workshop on Intelligent Process Automation (IPA 20). I hope you can come if you’ll be at AAAI 2020!
2019-11-26
📃 My thesis proposal document is available online, and the video of my talk is available too!
2019-08-01
📃 Our paper “PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations” has been accepted to UIST 2019!
2018-10-02
💬 My awesome summer intern Marissa Radensky will present our study How End Users Express Conditionals in Programming by Demonstration for Mobile Apps at VL/HCC 2018!
2018-07-31
📃 Our paper “APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions” has been accepted to VL/HCC 2018!
2018-04-22
💬 I will talk about our position paper “Supporting Co-adaptive Human-Agent Relationship through Programming by Demonstration using Existing GUIs” at CHI 2018 Workshop on Rethinking Interaction.
2018-02-28
📃 Our paper “KITE: Building conversational bots from mobile apps” has been accepted to MobiSys 2018!
2017-06-15
🏆 We have received a Best Paper award at IS-EUD 2017 for our paper “Programming IoT Devices by Demonstration Using Mobile Apps”. https://www.nd.edu/* 2017-03-22
📃 Our paper “Programming IoT Devices by Demonstration Using Mobile Apps” has been accepted to The 6th International Symposium on End User Development (IS-EUD 2017).
2017-02-24
💬 I will present our work on Designing a Conversational Interface for a Multimodal Smartphone Programming-by-Demonstration Agent at the Conversational UX Design CHI 2017 Workshop.
2017-02-17
🏅 We have received a Best Paper Honorable Mention award at CHI 2017 for our paper “SUGILITE: Creating Multimodal Smartphone Automation by Demonstration”.
2017-01-06
📃 Our paper “SUGILITE: Creating Multimodal Smartphone Automation by Demonstration” has been accepted to CHI ’17! A Video DEMO is also available.
2016-03-10
🏅 I’m honored to receive the Yahoo! Fellowship ($100,000/year worth) to support my research in automating cross-application tasks on smartphone.
2016-02-16
💬 I will present our ongoing work “Smartphone Text Entry in Cross-Application Tasks” on CHI 2016 Workshop on Inviscid Text Entry and Beyond at San Jose, CA on May 8, 2016
2015-04-15
🏫 Happy to announce that I’ll be joining HCII @ CMU as a PhD student in HCI starting next fall!

Travel

2025-10-18 – 2025-10-22
Bergen, Norway for CSCW 2025
2025-10-06 – 2025-10-10
Raleigh, NC for VL/HCC 2025
2025-09-28 – 2025-10-01
Busan, Korea for UIST 2025
2025-04-26 – 2025-05-01
Yokohama, Japan for CHI 2025
2025-03-24 – 2025-03-27
Cagliari, Italy for IUI 2025
2024-11-09 – 2024-11-13
San José, Costa Rica for CSCW 2024
2024-10-13 – 2024-10-16
Pittsburgh, PA for UIST 2024
2024-09-03 – 2024-09-06
Liverpool, UK for VL/HCC 2024
2024-06-23 – 2024-06-26
Chicago, IL for C&C 2024
2024-06-16 – 2024-06-20
Delavan, WI for HCIC 2024
2024-05-11 – 2024-05-16
Hawaii, HI for CHI 2024
2024-04-01 – 2024-04-03
Princeton, NJ for giving an Invited William Field Pierson Lecture
2023-12-06 – 2023-12-10
Singapore for EMNLP 2023
2023-10-29 – 2023-11-01
San Francisco, CA for UIST 2023
2023-10-13 – 2023-10-18
Minneapolis, MN for CSCW 2023
2023-07-25 – 2023-07-29
Honolulu, HI for ICML 2023
2023-04-23 – 2023-04-28
Hamburg, Germany for CHI 2023
2023-03-27 – 2023-03-31
Sydney, Australia for IUI 2023
2023-02-13 – 2023-02-14
Pittsburgh, PA for PLATEAU 2023
2022-06-08 – 2022-06-09
Durham, NH for CHIWORK 2022
2022-04-30 – 2022-05-06
New Orleans, LA for CHI 2022

MORE...

2022-02-23 – 2022-02-25
Washington, DC for CRA Career Mentoring Workshop
2021-11-08 – 2020-11-10
Pittsburgh, PA for PLATEAU 2021
2021-05-08 – 2021-05-13
~~Yokohama, Japan for CHI 2021~~ (Cancelled due to COVID-19)
2020-10-17 – 2020-10-23
~~Minneapolis, MN for CSCW 2020 and UIST 2020~~ (Cancelled due to COVID-19)
2020-07-05 – 2020-07-10
~~Seattle, WA for ACL 2020~~ (Cancelled due to COVID-19)
2020-04-25 – 2020-04-30
~~Honolulu, HI for CHI 2020~~ (Cancelled due to COVID-19)
2020-02-06 – 2020-02-09
New York, NY for AAAI 2020
2019-10-20 – 2019-10-23
New Orleans, LA for UIST 2019
2019-05-23 – 2019-05-24
New York, NY for giving a talk at J.P. Morgan
2019-05-04 – 2019-05-09
Glasgow, Scotland for CHI 2019
2019-03-17 – 2019-03-20
Los Angeles, CA for IUI 2019
2018-10-01 – 2018-10-04
Lisbon, Portugal for VL/HCC 2018
2018-06-24 – 2018-06-28
Pajaro Dunes, Watsonville, CA for HCIC 2018
2018-06-10 – 2018-06-15
Munich, Germany for MobiSys 2018
2018-05-29 – 2018-05-31
Sunnyvale, CA for giving a talk at Oath
2018-04-21 – 2018-04-26
Montreal, QC for CHI 2018
2018-01-08 – 2018-01-15
Ithaca, NY
2017-10-11 – 2017-10-14
Raleigh, NC for VL/HCC 2017
2017-06-13 – 2017-06-15
Eindhoven, The Netherlands for IS-EUD 2017
2017-05-22 – 2017-08-18
Redmond, WA – Intern at Microsoft Research
2017-05-06 – 2017-05-12
Denver, CO for CHI 2017
2016-12-30 – 2017-01-01
Ottawa, ON
2016-12-28 – 2016-12-30
Toronto, ON
2016-05-07 – 2016-05-12
San Jose, CA for CHI 2016
2014-11-04 – 2014-11-07
Dallas, TX for SIGSPATIAL 2014

Recent Publications

UIST 2025

GLITTER: An AI-assisted Platform for Material-Grounded Asynchronous Discussion in Flipped Learning
Weirui Peng*, Yinuo Yang*, Zheng Zhang, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2025)
[Paper PDF]

UIST 2025

AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and Videos
Zheng Ning, Leyang Li, Daniel Killough, JooYoung Seo, Patrick Carrington, Yapeng Tian, Yuhang Zhao, Franklin Mingzhe Li, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2025)
[Paper PDF]

UIST 2025

Why am I seeing this: Democratizing End User Auditing for Online Content Recommendations
Chaoran Chen, Leyang Li, Luke Cao, Yanfang Ye, Tianshi Li, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2025)
[Paper PDF]

ACL Findings 2025

Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning
Simret Araya Gebreegziabher, Kuangshi Ai, Zheng Zhang, Elena L. Glassman, and Toby Jia-Jun Li
Findings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL Findings 2025)
[Paper PDF]

ACL Findings 2025

Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents
Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu, Toby Jia-Jun Li, and Dakuo Wang
Findings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL Findings 2025)
[Paper PDF]

CHIWORK 2025

MetricMate: An Interactive Tool for Generating Evaluation Criteria for LLM-as-a-Judge Workflow
Simret Araya Gebreegziabher, Yukun Yang, Elena L. Glassman, and Toby Jia-Jun Li
Proceedings of the 3rd Symposium on Human-Computer Interaction for Work (CHIWORK 2025)
[Paper PDF] [ACM DL]

CHI 2025

Supporting Co-Adaptive Machine Teaching through Human Concept Learning and Cognitive Theories
Simret Araya Gebreegziabher, Yukun Yang, Elena L. Glassman, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
🏆 Best Paper Award
[Paper PDF] [ACM DL]

CHI 2025

LADICA: A Large Shared Display Interface for Generative AI Cognitive Assistance in Co-located Team Collaboration
Zheng Zhang, Weirui Peng, Xinyue Chen, Luke Cao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

CHI 2025

Hashtag Re-Appropriation for Audience Control on Recommendation-Driven Social Media Xiaohongshu (rednote)
Ruyuan Wan, Lingbo Tong, Tiffany Knearem, Toby Jia-Jun Li, Ting-Hao Kenneth Huang, and Qunfang Wu
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL]

CHI 2025

From Operation to Cognition: Automatic Modeling Cognitive Dependencies from User Demonstrations for GUI Task Automation
Yiwen Yin, Yu Mei, Chun Yu, Toby Jia-Jun Li, Aamir Khan Jadoon, Sixiang Cheng, Weinan Shi, Mohan Chen, and Yuanchun Shi
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
[Paper PDF] [ACM DL]

IUI 2025

CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applications
Chaoran Chen, Daodao Zhou, Yanfang Ye, Toby Jia-Jun Li, and Yaxing Yao
Proceedings of the 30th ACM Conference on Intelligent User Interfaces (IUI 2025)
[Paper PDF] [ACM DL]

IUI 2025

Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer
Proceedings of the 30th ACM Conference on Intelligent User Interfaces (IUI 2025)
[Paper PDF] [ACM DL]

IUI 2025

Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models
Shiran Dudy, Thulasi Tholeti, Resmi Ramachandranpillai, Ali Muhammad, Toby Jia-Jun Li, and Ricardo Baeza-Yates
Proceedings of the 30th ACM Conference on Intelligent User Interfaces (IUI 2025)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL]

CSCW 2025

I’m categorizing LLM as a productivity tool: Examining ethics of LLM use in HCI research practices
Shivani Kapania, Ruiyi Wang, Toby Jia-Jun Li, Tianshi Li, and Hong Shen
Proceedings of the ACM on Human-Computer Interaction (CSCW 2025)
[Paper PDF] [ACM DL]

NDSS 2025

Careful About What App Promotion Ads Recommend! Detecting and Explaining Malware Promotion via App Promotion Graph
Shang Ma, Chaoran Chen, Shao Yang, Shifu Hou, Toby Jia-Jun Li, Xusheng Xiao, Tao Xie, and Yanfang Ye
The Network and Distributed System Security Symposium (NDSS 2025)
[Paper PDF]

CHI PLAY 2024

Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AI
Long Ling, Xinyi Chen, Ruoyu Wen, Toby Jia-Jun Li, and Ray LC
Proceedings of the ACM on Human-Computer Interaction (CHI PLAY 2024)
[Paper PDF] [ACM DL]

UIST 2024

SQLucid: Grounding Natural Language Database Queries with Interactive Explanations
Yuan Tian, Jonathan Kummerfeld, Toby Jia-Jun Li, and Tianyi Zhang
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2024)
[Paper PDF] [ACM DL]

C&C 2024

MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos
Zheng Ning, Zheng Zhang, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, and Toby Jia-Jun Li
Proceedings of the 16th ACM Conference on Creativity and Cognition (C&C 2024)
[Paper PDF] [ACM DL]

FSE 2024

EyeTrans: Merging Human and Machine Attention for Neural Code Summarization
Yifan Zhang, Jiliang Li, Zachary Karas, Aakash Bansal, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, and Yu Huang
Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024)
[Paper PDF] [ACM DL]

CHI 2024

An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors
Chaoran Chen, Weijun Li, Wenxin Song, Yanfang Ye, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video]

CHI 2024

SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers
Zheng Ning, Brianna L. Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video]

CHI 2024

Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation
Sangho Suh*, Meng Chen*, Bryan Min, Toby Jia-Jun Li, and Haijun Xia
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video] [Website]

CHI 2024

CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models
Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Perrault
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video]

MobiCom 2024

AutoDroid: LLM-powered Task Automation in Android
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu
Proceedings of the 30th Annual International Conference on Mobile Computing and Networking (MobiCom 2024)
[Paper PDF]

CSCW 2024

From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX
Yuwen Lu, Chao Zhang, Yuewen Yang, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM on Human-Computer Interaction (CSCW 2024)
🏆 Best Paper Award
[Paper PDF] [ACM DL]

EMNLP 2023

Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations
Yuan Tian, Zheng Zhang, Zheng Ning, Toby Jia-Jun Li, Jonathan K. Kummerfeld, and Tianyi Zhang
Proceedings of the the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)
[Paper PDF]

UIST 2023

VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping
Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2023)
[Paper PDF] [ACM DL] [Demo Video] [Website]

UIST 2023

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
Zheng Zhang*, Zheng Ning*, Chenliang Xu, Yapeng Tian, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2023)
[Paper PDF] [ACM DL] [Demo Video]

CHI 2023

PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis
Simret Araya Gebreegziabher*, Zheng Zhang*, Xiaohang Tang, Yihao Meng, Elena Glassman, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2023)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

IUI 2023

An Empirical Study of Model Errors and User Error Discovery and Repair Strategies in Natural Language Database Queries
Zheng Ning*, Zheng Zhang*, Tianyi Sun, Yuan Tian, Tianyi Zhang, and Toby Jia-Jun Li
Proceedings of the 28th ACM Conference on Intelligent User Interfaces (IUI 2023)
[Paper PDF] [ACM DL]

CHIWORK 2022

A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality
Toby Jia-Jun Li, Yuwen Lu, Jaylexia Clark, Meng Chen, Victor Cox, Meng Jiang, Yang Yang, Tamara Kay, Danielle Wood, and Jay Brockman
Proceedings of the 1st Symposium on Human-Computer Interaction for Work (CHIWORK 2022)
[Paper PDF] [ACM DL]

ACL 2022

It is AI’s Turn to Ask Human a Question: Question and Answer Pair Generation for Children Storybooks in FairytaleQA Dataset
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Mo Yu, and Ying Xu
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)
[Paper PDF] [ACL Anthology]

ACL 2022

Fantastic Questions and Where to Find Them: FairytaleQA–An Authentic Dataset for Narrative Comprehension
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)
[Paper PDF] [ACL Anthology]

CHI 2022

StoryBuddy: A Human-AI Collaborative Agent for Parent-Child Interactive Storytelling with Flexible Parent Involvement
Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI 2022)
[Paper PDF] [ACM DL] [Demo Video]

CHI 2021

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [GitHub]

UIST 2020

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, and Brad A. Myers
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
🏆 Best Paper Award
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

UIST 2020

Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
Ritam Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang ‘Anthony’ Chen
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
[Paper PDF] [ACM DL] [Demo Video]

CSCW 2020

Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, and Brad A. Myers
Proceedings of the ACM on Human-Computer Interaction (CSCW 2020)
[Paper PDF] [ACM DL]

ACL 2020 Demo

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
Toby Jia-Jun Li, Tom Mitchell, and Brad A. Myers
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020): System Demonstrations
[Paper PDF] [ACL Anthology] [Talk Video]

UIST 2019

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST 2019)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

VL/HCC 2018

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018)
[Paper PDF] [IEEE Xplore] [Demo Video]

MobiSys 2018

KITE: Building conversational bots from mobile apps
Toby Jia-Jun Li and Oriana Riva
Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018)
[Paper PDF] [ACM DL] [Talk Video]

IMWUT/Ubicomp 2017

PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps
Yuanchun Li, Fanglin Chen, Toby Jia-jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, and Jason I. Hong
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2017)
[Paper PDF] [ACM DL] [Website]

IS-EUD 2017

Programming IoT Devices by Demonstration Using Mobile Apps
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A. Myers
Proceedings of the International Symposium on End User Development (IS-EUD 2017)
🏆 Best Paper Award
[Paper PDF] [SpringerLink]

CHI 2017

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

CHI 2016

Not at Home on the Range: Peer Production and the Urban/Rural Divide
Isaac Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016)
[Paper PDF] [ACM DL]

SIGSPATIAL 2014

Leveraging Advances in Natural Language Processing to Better Understand Tobler’s First Law of Geography
Toby Jia-Jun Li, Shilad Sen, and Brent Hecht
Proceedings of the ACM Conference on Advances in Geographic Information Systems (SIGSPATIAL 2014)
[Paper PDF] [ACM DL]

OpenSym / WikiSym 2014

WikiBrain: Democratizing Computation on Wikipedia
Shilad Sen, Toby Jia-Jun Li, WikiBrain Team, and Brent Hecht
Proceedings of the 10th International Symposium on Open Collaboration (OpenSym / WikiSym 2014)
[Paper PDF] [ACM DL]