Publications

Stayin’ Aligned Over Time: Toward Longitudinal Human-LLM Alignment via Contextual Reflection and Privacy-Preserving Behavioral Data
Simret Araya Gebreegziabher, Allison Sproul, Yinuo Yang, Chaoran Chen, Diego Gómez-Zará, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2026)
[Paper PDF]

NaturalEdit: Code Modification through Direct Interaction with Adaptive Natural Language Representation
Ningzhi Tang, David Meininger, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2026)
[Paper PDF]

How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, and Toby Jia-Jun Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)
[Paper PDF]

Humans’ ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model ANnotations for Agent Collaboration
Jiaju Chen, Yuxuan Lu, Jiayi Su, Chaoran Chen, Songlin Xiao, Zheng Zhang, Yun Wang, Yunyao Li, Jian Zhao, Tongshuang Wu, Toby Jia-Jun Li, Dakuo Wang, and Bingsheng Yao
Findings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP Findings 2026)
[Paper PDF]

Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions
Ningzhi Tang, Chaoran Chen, Zihan Fang, Gelei Xu, Maria Dhakal, Yiyu Shi, Collin McMillan, Yu Huang, and Toby Jia-Jun Li
Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)
[Paper PDF]

MultEval: Supporting Collaborative Alignment for LLM-as-a-Judge Evaluation Criteria
Charles Chiang, Simret Gebreegziabher, Annalisa Szymanski, Hyo Jin Do, Zahra Ashktorab, Werner Geyer, Toby Jia-Jun Li, and Diego Gómez-Zará
Proceedings of the 5th Symposium on Human-Computer Interaction for Work (CHIWORK 2026)
[Paper PDF] [ACM DL]

Uncovering Disparities in Rideshare Drivers’ Earning and Work Patterns: A Case Study of Chicago
Hy Dang, Yuwen Lu, Jason Spicer, Tamara Kay, Di Yang, Yang Yang, Jay Brockman, Meng Jiang, and Toby Jia-Jun Li
Proceedings of the ACM on Human-Computer Interaction (CSCW 2026)
[Paper PDF]

OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation
Ziyi Wang, Yuxuan Lu, Wenbo Li, Amirali Amini, Bo Sun, Yakov Bart, Weimin Lyu, Jiri Gesi, Tian Wang, Jing Huang, Yu Su, Upol Ehsan, Malihe Alikhani, Toby Jia-Jun Li, Lydia Chilton, and Dakuo Wang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
[Paper PDF] [ACL Anthology]

EyeMulator: Improving Code Language Models by Mimicking Human Visual Attention
Yifan Zhang, Chen Huang, Yueke Zhang, Jiahao Zhang, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, and Yu Huang
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
[Paper PDF] [ACL Anthology]

PrivacyMotiv: Vulnerability-Centered Persona Journeys for Empathic Privacy Reviews in UX Design
Zeya Chen, Jianing Wen, Yaxing Yao, Toby Jia-Jun Li, and Tianshi Li
Proceedings of the 2026 ACM Designing Interactive Systems Conference (DIS 2026)
[Paper PDF] [ACM DL]

Brushing and Interpretive Linking for Knowledge Work
Zheng Ning, Michel Pahud, Hugo Romat, Nicolai Marquardt, Fanny Chevalier, Jeevana Priya Inala, Chenglong Wang, David Brown, Toby Jia-Jun Li, Ken Hinckley, and Nathalie Riche
Proceedings of Graphics Interface 2026 (GI 2026)
[Paper PDF]

Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight
Jingyu Tang, Chaoran Chen, Jiawen Li, Zhiping Zhang, Bingcan Guo, Ibrahim Khalilov, Simret Araya Gebreegziabher, Bingsheng Yao, Dakuo Wang, Yanfang Ye, Tianshi Li, Ziang Xiao, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2026)
[Paper PDF] [ACM DL]

Vistoria: A Multimodal System to Support Fictional Story Writing through Instrumental Text–Image Co-Editing
Kexue Fu, Jingfei Huang, Long Ling, Sumin Hong, Yihang Zuo, Ray LC, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2026)
[Paper PDF] [ACM DL]

Crepe: A Mobile Screen Data Collector Using Graph Query
Yuwen Lu, Meng Chen, Qi Zhao, Victor Cox IV, Yang Yang, Meng Jiang, Jay Brockman, Tamara Kay, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2026)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL]

Designing Staged Evaluation Workflows for LLMs: Integrating Domain Experts, Lay Users, and Model-Generated Evaluation Criteria
Annalisa Szymanski, Simret Araya Gebreegziabher, Maro Anuyah, Ronald Metoyer, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2026)
[Paper PDF] [ACM DL]

Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration
Bingsheng Yao, Jiaju Chen, Chaoran Chen, April Yi Wang, Toby Jia-Jun Li, and Dakuo Wang
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2026)
[Paper PDF] [ACM DL]

My Favorite Streamer is an LLM: Discovering, Bonding, and Co-Creating in AI VTuber Fandom
Jiayi Ye, Chaoran Chen, Yue Huang, Yanfang Ye, Toby Jia-Jun Li, and Xiangliang Zhang
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2026)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL]

The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes
Simret Araya Gebreegziabher, Yukun Yang, Charles Chiang, Hojun Yoo, Chaoran Chen, Hyo Jin Do, Zahra Ashktorab, Werner Geyer, Diego Gómez-Zará, and Toby Jia-Jun Li
Proceedings of the 31st ACM Conference on Intelligent User Interfaces (IUI 2026)
[Paper PDF] [ACM DL]

Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study
Annalisa Szymanski, Oghenemaro Anuyah, Toby Jia-Jun Li, and Ronald Metoyer
Proceedings of the 31st ACM Conference on Intelligent User Interfaces (IUI 2026)
[Paper PDF] [ACM DL]

Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification
Ningzhi Tang, Emory Smith, Yu Huang, Collin McMillan, and Toby Jia-Jun Li
Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2025)
[Paper PDF] [IEEE Xplore]

GLITTER: An AI-assisted Platform for Material-Grounded Asynchronous Discussion in Flipped Learning
Weirui Peng*, Yinuo Yang*, Zheng Zhang, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2025)
[Paper PDF] [ACM DL]

AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and Videos
Zheng Ning, Leyang Li, Daniel Killough, JooYoung Seo, Patrick Carrington, Yapeng Tian, Yuhang Zhao, Franklin Mingzhe Li, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2025)
[Paper PDF] [ACM DL]

Why am I seeing this: Democratizing End User Auditing for Online Content Recommendations
Chaoran Chen, Leyang Li, Luke Cao, Yanfang Ye, Tianshi Li, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2025)
[Paper PDF] [ACM DL]

Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning
Simret Araya Gebreegziabher, Kuangshi Ai, Zheng Zhang, Elena L. Glassman, and Toby Jia-Jun Li
Findings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL Findings 2025)
[Paper PDF] [ACL Anthology]

Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agents
Chaoran Chen, Bingsheng Yao, Ruishi Zou, Wenyue Hua, Weimin Lyu, Toby Jia-Jun Li, and Dakuo Wang
Findings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL Findings 2025)
[Paper PDF] [ACL Anthology]

MetricMate: An Interactive Tool for Generating Evaluation Criteria for LLM-as-a-Judge Workflow
Simret Araya Gebreegziabher, Charles Chiang, Zichu Wang, Zahra Ashktorab, Michelle Brachman, Werner Geyer, Toby Jia-Jun Li, and Diego Gómez-Zará
Proceedings of the 3rd Symposium on Human-Computer Interaction for Work (CHIWORK 2025)
[Paper PDF] [ACM DL]

Supporting Co-Adaptive Machine Teaching through Human Concept Learning and Cognitive Theories
Simret Araya Gebreegziabher, Yukun Yang, Elena L. Glassman, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
🏆 Best Paper Award
[Paper PDF] [ACM DL]

LADICA: A Large Shared Display Interface for Generative AI Cognitive Assistance in Co-located Team Collaboration
Zheng Zhang, Weirui Peng, Xinyue Chen, Luke Cao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

Hashtag Re-Appropriation for Audience Control on Recommendation-Driven Social Media Xiaohongshu (rednote)
Ruyuan Wan, Lingbo Tong, Tiffany Knearem, Toby Jia-Jun Li, Ting-Hao Kenneth Huang, and Qunfang Wu
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL]

From Operation to Cognition: Automatic Modeling Cognitive Dependencies from User Demonstrations for GUI Task Automation
Yiwen Yin, Yu Mei, Chun Yu, Toby Jia-Jun Li, Aamir Khan Jadoon, Sixiang Cheng, Weinan Shi, Mohan Chen, and Yuanchun Shi
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2025)
[Paper PDF] [ACM DL]

CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applications
Chaoran Chen, Daodao Zhou, Yanfang Ye, Toby Jia-Jun Li, and Yaxing Yao
Proceedings of the 30th ACM Conference on Intelligent User Interfaces (IUI 2025)
[Paper PDF] [ACM DL]

Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer
Proceedings of the 30th ACM Conference on Intelligent User Interfaces (IUI 2025)
[Paper PDF] [ACM DL]

Unequal Opportunities: Examining the Bias in Geographical Recommendations by Large Language Models
Shiran Dudy, Thulasi Tholeti, Resmi Ramachandranpillai, Ali Muhammad, Toby Jia-Jun Li, and Ricardo Baeza-Yates
Proceedings of the 30th ACM Conference on Intelligent User Interfaces (IUI 2025)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL]

I’m categorizing LLM as a productivity tool: Examining ethics of LLM use in HCI research practices
Shivani Kapania, Ruiyi Wang, Toby Jia-Jun Li, Tianshi Li, and Hong Shen
Proceedings of the ACM on Human-Computer Interaction (CSCW 2025)
[Paper PDF] [ACM DL]

Careful About What App Promotion Ads Recommend! Detecting and Explaining Malware Promotion via App Promotion Graph
Shang Ma, Chaoran Chen, Shao Yang, Shifu Hou, Toby Jia-Jun Li, Xusheng Xiao, Tao Xie, and Yanfang Ye
The Network and Distributed System Security Symposium (NDSS 2025)
[Paper PDF]

Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AI
Long Ling, Xinyi Chen, Ruoyu Wen, Toby Jia-Jun Li, and Ray LC
Proceedings of the ACM on Human-Computer Interaction (CHI PLAY 2024)
[Paper PDF] [ACM DL]

SQLucid: Grounding Natural Language Database Queries with Interactive Explanations
Yuan Tian, Jonathan Kummerfeld, Toby Jia-Jun Li, and Tianyi Zhang
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2024)
[Paper PDF] [ACM DL]

MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos
Zheng Ning, Zheng Zhang, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, and Toby Jia-Jun Li
Proceedings of the 16th ACM Conference on Creativity and Cognition (C&C 2024)
[Paper PDF] [ACM DL]

EyeTrans: Merging Human and Machine Attention for Neural Code Summarization
Yifan Zhang, Jiliang Li, Zachary Karas, Aakash Bansal, Toby Jia-Jun Li, Collin McMillan, Kevin Leach, and Yu Huang
Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE 2024)
[Paper PDF] [ACM DL]

An Empathy-Based Sandbox Approach to Bridge the Privacy Gap among Attitudes, Goals, Knowledge, and Behaviors
Chaoran Chen, Weijun Li, Wenxin Song, Yanfang Ye, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video]

SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers
Zheng Ning, Brianna L. Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video]

Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation
Sangho Suh*, Meng Chen*, Bryan Min, Toby Jia-Jun Li, and Haijun Xia
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video] [Website]

CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models
Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Perrault
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2024)
[Paper PDF] [ACM DL] [Demo Video]

AutoDroid: LLM-powered Task Automation in Android
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu
Proceedings of the 30th Annual International Conference on Mobile Computing and Networking (MobiCom 2024)
[Paper PDF]

From Awareness to Action: Exploring End-User Empowerment Interventions for Dark Patterns in UX
Yuwen Lu, Chao Zhang, Yuewen Yang, Yaxing Yao, and Toby Jia-Jun Li
Proceedings of the ACM on Human-Computer Interaction (CSCW 2024)
🏆 Best Paper Award
[Paper PDF] [ACM DL]

Interactive Text-to-SQL Generation via Editable Step-by-Step Explanations
Yuan Tian, Zheng Zhang, Zheng Ning, Toby Jia-Jun Li, Jonathan K. Kummerfeld, and Tianyi Zhang
Proceedings of the the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)
[Paper PDF]

VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping
Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2023)
[Paper PDF] [ACM DL] [Demo Video] [Website]

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
Zheng Zhang*, Zheng Ning*, Chenliang Xu, Yapeng Tian, and Toby Jia-Jun Li
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2023)
[Paper PDF] [ACM DL] [Demo Video]

PaTAT: Human-AI Collaborative Qualitative Coding with Explainable Interactive Rule Synthesis
Simret Araya Gebreegziabher*, Zheng Zhang*, Xiaohang Tang, Yihao Meng, Elena Glassman, and Toby Jia-Jun Li
Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2023)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

An Empirical Study of Model Errors and User Error Discovery and Repair Strategies in Natural Language Database Queries
Zheng Ning*, Zheng Zhang*, Tianyi Sun, Yuan Tian, Tianyi Zhang, and Toby Jia-Jun Li
Proceedings of the 28th ACM Conference on Intelligent User Interfaces (IUI 2023)
[Paper PDF] [ACM DL]

A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality
Toby Jia-Jun Li, Yuwen Lu, Jaylexia Clark, Meng Chen, Victor Cox, Meng Jiang, Yang Yang, Tamara Kay, Danielle Wood, and Jay Brockman
Proceedings of the 1st Symposium on Human-Computer Interaction for Work (CHIWORK 2022)
[Paper PDF] [ACM DL]

It is AI’s Turn to Ask Human a Question: Question and Answer Pair Generation for Children Storybooks in FairytaleQA Dataset
Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Mo Yu, and Ying Xu
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)
[Paper PDF] [ACL Anthology]

Fantastic Questions and Where to Find Them: FairytaleQA–An Authentic Dataset for Narrative Comprehension
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022)
[Paper PDF] [ACL Anthology]

StoryBuddy: A Human-AI Collaborative Agent for Parent-Child Interactive Storytelling with Flexible Parent Involvement
Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI 2022)
[Paper PDF] [ACM DL] [Demo Video]

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [GitHub]

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, and Brad A. Myers
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
🏆 Best Paper Award
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
Ritam Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang ‘Anthony’ Chen
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
[Paper PDF] [ACM DL] [Demo Video]

Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, and Brad A. Myers
Proceedings of the ACM on Human-Computer Interaction (CSCW 2020)
[Paper PDF] [ACM DL]

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
Toby Jia-Jun Li, Tom Mitchell, and Brad A. Myers
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020): System Demonstrations
[Paper PDF] [ACL Anthology] [Talk Video]

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST 2019)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018)
[Paper PDF] [IEEE Xplore] [Demo Video]

KITE: Building conversational bots from mobile apps
Toby Jia-Jun Li and Oriana Riva
Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018)
[Paper PDF] [ACM DL] [Talk Video]

PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps
Yuanchun Li, Fanglin Chen, Toby Jia-jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, and Jason I. Hong
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2017)
[Paper PDF] [ACM DL] [Website]

Programming IoT Devices by Demonstration Using Mobile Apps
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A. Myers
Proceedings of the International Symposium on End User Development (IS-EUD 2017)
🏆 Best Paper Award
[Paper PDF] [SpringerLink]

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

Not at Home on the Range: Peer Production and the Urban/Rural Divide
Isaac Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016)
[Paper PDF] [ACM DL]

Leveraging Advances in Natural Language Processing to Better Understand Tobler’s First Law of Geography
Toby Jia-Jun Li, Shilad Sen, and Brent Hecht
Proceedings of the ACM Conference on Advances in Geographic Information Systems (SIGSPATIAL 2014)
[Paper PDF] [ACM DL]

WikiBrain: Democratizing Computation on Wikipedia
Shilad Sen, Toby Jia-Jun Li, WikiBrain Team, and Brent Hecht
Proceedings of the 10th International Symposium on Open Collaboration (OpenSym / WikiSym 2014)
[Paper PDF] [ACM DL]

Demonstration + Natural Language: Multimodal Interfaces for GUI-based Interactive Task Learning Agents
Toby Jia-Jun Li, Tom M. Mitchell, and Brad A. Myers
Chapter of Artificial Intelligence for Human Computer Interaction: A Modern Approach., Springer, 2021
[Paper PDF] [SpringerLink]

Teaching Agents When They Fail: End User Development in Goal-Oriented Conversational Agents
Toby Jia-Jun Li, Igor Labutov, Brad A. Myers, Amos Azaria, Alexander I. Rudnicky, and Tom M. Mitchell
Chapter of Studies in Conversational UX Design, Springer, 2018
[SpringerLink]

Making End User Development More Natural
Brad A. Myers, Amy Ko, Chris Scaffidi, Stephen Oney, YoungSeok Yoon, Kerry Chang, Mary Beth Kery, and Toby Jia-Jun Li
Chapter of New Perspectives in End-User Development, Springer, 2017
[SpringerLink]

Automatically generating conversational services from a computing application
Oriana Riva, Jason Kace, Doug Burger, and Toby Jia-Jun Li
U.S. Patent Application 16/002,915. Filed December 12, 2019
[Google Patents]

An Empirical Study of Developer Behaviors for Validating and Repairing AI-Generated Code
Ningzhi Tang*, Meng Chen*, Zheng Ning, Aakash Bansal, Yu Huang, Collin McMillan, and Toby Jia-Jun Li
The 13th Annual Workshop on the Intersection of PL and HCI (PLATEAU 2023)
[Paper PDF]

SHAI 2023: Workshop on Designing for Safety in Human-AI Interactions
Nitesh Goyal, Sungsoo Ray Hong, Regan Mandryk, Toby Jia-Jun Li, Kurt Luther, and Dakuo Wang
Extended Abstracts of the 2023 ACM Conference on Human Factors in Computing Systems (CHI EA ’23)
[Paper PDF]

The Future of Computational Approaches for Understanding and Adapting User Interfaces
Yue Jiang, Yuwen Lu, Christof Lutteroth, Toby Jia-Jun Li, Jeffrey Nichols, and Wolfgang Stuerzlinger
Extended Abstracts of the 2023 ACM Conference on Human Factors in Computing Systems (CHI EA ’23)
[Paper PDF]

MIMOSA: Human-in-the-Loop Generation of Spatial Audio from Videos with Monaural Audio
Zheng Ning*, Zheng Zhang*, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, and Toby Jia-Jun Li
ECCV 2022 Workshop on Visual Learning of Sounds in Spaces (AV4D)
[Paper PDF]

Computational Approaches for Understanding, Generating, and Adapting User Interfaces
Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, and Toby Jia-Jun Li
Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22)
[Paper PDF]

Bridging the Gap Between UX Practitioners’ Work Practices and Machine-Learning-Enabled Design Support Tools
Yuwen Lu, Chengzhi Zhang, Iris Zhang, and Toby Jia-Jun Li
Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22)
[Paper PDF]

AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing
Daijin Yang, Yanpeng Zhou, Zhiyuan Zhang, Toby Jia-Jun Li, and Ray LC
IUI 2022 Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN 2022)
[Paper PDF]

Building an Interactive Storytelling Conversational Agent through Parent-AI Collaboration
Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li
CSCW 2021 Workshop on Inclusive and Collaborative Child-facing Voice Technologies (CUI@CSCW)
[Paper PDF]

Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents
Toby Jia-Jun Li, Jingya Chen, Tom M. Mitchell, and Brad A. Myers
CHI 2020 Workshop on Artificial Intelligence for HCI: A Modern Approach (AI4HCI)
[Paper PDF]

Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
The AAAI-20 Workshop on Intelligent Process Automation (IPA-20)
[Paper PDF]

A Multi-Modal Approach to Concept Learning in Task Oriented Conversational Agents
Toby Jia-Jun Li, Marissa Radensky, Tom M. Mitchell, and Brad A. Myers
CHI 2019 Workshop on Conversational Agents: Acting on the Wave of Research and Development
[Paper PDF]

How End Users Express Conditionals in Programming by Demonstration for Mobile Apps
Marissa Radensky, Toby Jia-Jun Li, and Brad A. Myers
2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018) Poster Track
[Paper PDF] [IEEE Xplore]

Supporting Co-adaptive Human-Agent Relationship through Programming by Demonstration using Existing GUIs
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Tom M. Mitchell, and Brad A. Myers
Rethinking Interaction CHI 2018 Workshop
[Paper PDF]

End User Mobile Task Automation using Multimodal Programming by Demonstration
Toby Jia-Jun Li
IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2017) Graduate Consortium
[Paper PDF] [IEEE Xplore]

Designing a Conversational Interface for a Multimodal Smartphone Programming-by-Demonstration Agent
Toby Jia-Jun Li, Brad A. Myers, Amos Azaria, Igor Labutov, Alexander I. Rudnicky, and Tom M. Mitchell
Conversational UX Design CHI 2017 Workshop
[Paper PDF]

Smartphone Text Entry in Cross-Application Tasks
Toby Jia-Jun Li and Brad A. Myers
CHI 2016 Workshop on Inviscid Text Entry and Beyond
[Paper PDF]