Publications

Major Peer-reviewed Conference and Journal Papers

CHI 2021

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021)
[Paper PDF] [GitHub]

UIST 2020

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, and Brad A. Myers
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
🏆 Best Paper Award
[Paper PDF] [Demo Video] [Talk Video]

UIST 2020

Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
Ritam Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang ‘Anthony’ Chen
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
[Paper PDF] [Demo Video]

CSCW 2020

Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, and Brad A. Myers
Proceedings of the ACM on Human-Computer Interaction (CSCW 2020)
[Paper PDF] [ACM DL]

ACL 2020 Demo

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
Toby Jia-Jun Li, Tom Mitchell, and Brad A. Myers
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020): System Demonstrations
[Paper PDF] [ACL Anthology] [Talk Video]

UIST 2019

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST 2019)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

VL/HCC 2018

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018)
[Paper PDF] [IEEE Xplore] [Demo Video]

MobiSys 2018

KITE: Building conversational bots from mobile apps
Toby Jia-Jun Li and Oriana Riva
Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018)
[Paper PDF] [ACM DL] [Talk Video]

IMWUT/Ubicomp 2017

PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps
Yuanchun Li, Fanglin Chen, Toby Jia-jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, and Jason I. Hong
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2017)
[Paper PDF] [ACM DL] [Website]

IS-EUD 2017

Programming IoT Devices by Demonstration Using Mobile Apps
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A. Myers
Proceedings of the International Symposium on End User Development (IS-EUD 2017)
🏆 Best Paper Award
[Paper PDF] [SpringerLink]

CHI 2017

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

CHI 2016

Not at Home on the Range: Peer Production and the Urban/Rural Divide
Isaac Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016)
[Paper PDF] [ACM DL]

SIGSPATIAL 2014

Leveraging Advances in Natural Language Processing to Better Understand Tobler’s First Law of Geography
Toby Jia-Jun Li, Shilad Sen, and Brent Hecht
Proceedings of the ACM Conference on Advances in Geographic Information Systems (SIGSPATIAL 2014)
[Paper PDF] [ACM DL]

OpenSym / WikiSym 2014

WikiBrain: Democratizing Computation on Wikipedia
Shilad Sen, Toby Jia-Jun Li, WikiBrain Team, and Brent Hecht
Proceedings of the 10th International Symposium on Open Collaboration (OpenSym / WikiSym 2014)
[Paper PDF] [ACM DL]

Book Sections

Teaching Agents When They Fail: End User Development in Goal-Oriented Conversational Agents
Toby Jia-Jun Li, Igor Labutov, Brad A. Myers, Amos Azaria, Alexander I. Rudnicky, and Tom M. Mitchell
Chapter of Studies in Conversational UX Design, Springer, 2018
[SpringerLink]

Making End User Development More Natural
Brad A. Myers, Amy Ko, Chris Scaffidi, Stephen Oney, YoungSeok Yoon, Kerry Chang, Mary Beth Kery, and Toby Jia-Jun Li
Chapter of New Perspectives in End-User Development, Springer, 2017
[SpringerLink]

Patents

Automatically generating conversational services from a computing application
Oriana Riva, Jason Kace, Doug Burger, and Toby Jia-Jun Li
U.S. Patent Application 16/002,915. Filed December 12, 2019
[Google Patents]

Minor Lightly-Reviewed Posters, Extended Abstracts, and Workshop Papers

Towards Effective Human-AI Collaboration in GUI-Based Interactive Task Learning Agents
Toby Jia-Jun Li, Jingya Chen, Tom M. Mitchell, and Brad A. Myers
CHI 2020 Workshop on Artificial Intelligence for HCI: A Modern Approach (AI4HCI)
[Paper PDF]

Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
The AAAI-20 Workshop on Intelligent Process Automation (IPA-20)
[Paper PDF]

A Multi-Modal Approach to Concept Learning in Task Oriented Conversational Agents
Toby Jia-Jun Li, Marissa Radensky, Tom M. Mitchell, and Brad A. Myers
CHI 2019 Workshop on Conversational Agents: Acting on the Wave of Research and Development
[Paper PDF]

How End Users Express Conditionals in Programming by Demonstration for Mobile Apps
Marissa Radensky, Toby Jia-Jun Li, and Brad A. Myers
2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018) Poster Track
[Paper PDF] [IEEE Xplore]

Supporting Co-adaptive Human-Agent Relationship through Programming by Demonstration using Existing GUIs
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Tom M. Mitchell, and Brad A. Myers
Rethinking Interaction CHI 2018 Workshop
[Paper PDF]

End User Mobile Task Automation using Multimodal Programming by Demonstration
Toby Jia-Jun Li
IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2017) Graduate Consortium
[Paper PDF] [IEEE Xplore]

Designing a Conversational Interface for a Multimodal Smartphone Programming-by-Demonstration Agent
Toby Jia-Jun Li, Brad A. Myers, Amos Azaria, Igor Labutov, Alexander I. Rudnicky, and Tom M. Mitchell
Conversational UX Design CHI 2017 Workshop
[Paper PDF]

Smartphone Text Entry in Cross-Application Tasks
Toby Jia-Jun Li and Brad A. Myers
CHI 2016 Workshop on Inviscid Text Entry and Beyond
[Paper PDF]

Invited Talks and Presentation

Interactive Systems for Configuring, Extending, and Developing AI Applications
Invited Talk at Google People + AI Research (PAIR)
Virtual Visit. Oct. 13, 2020

Interactive Systems for Configuring, Extending, and Developing AI Applications
Invited Talk at IBM Research Cambridge
Virtual Visit. Aug. 12, 2020

SUGILITE: A Multi-Modal Agent that Learns Tasks from Natural Language and Demonstrations
Demo Presentation at the ACL 2020 Workshop on Natural Language Interfaces
Seattle, WA. July 10, 2020

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
Invited Talk at The AAAI-20 Workshop on Intelligent Process Automation (IPA-20)
New York, NY. Feb 7, 2020

SUGILITE: A Multi-Modal Agent that Learns Tasks from Natural Language and Demonstrations
Invited Demo at The 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2019)
Pittsburgh, PA. Oct. 28, 2019

SUGILITE: A Multi-Modal Agent that Learns Tasks from Natural Language and Demonstrations
Invited Lightning Talk at CMU HCII 25th Anniversary
Pittsburgh, PA. Oct. 25, 2019

Machine Learning from Human Instruction: Every Person a Programmer
Invited Talk at J.P. Morgan (with Forough Arabshahi)
New York, NY. May 24, 2019

Teaching Intelligent Agents New Tricks: Natural Language Instructions plus Programming-by-Demonstration for Teaching Tasks
Invited Talk at Human Computer Interaction Consortium (HCIC ‘18) (with Forough Arabshahi)
Watsonville, CA. Jun. 25, 2018

SUGILITE: Enabling InMind Agent to Learn New Tasks from User Demonstration
Invited Talk at Oath (formerly Yahoo!)
Sunnyvale, CA. May 30, 2018

Atlasify – The Geography of Everything
Invited Demo at 3M Science and Engineering Symposium
St Paul, MN. Jun 25, 2015

Atlasify – The Geography of Everything
Invited Demo at The Social Media and Business Analytics Collaborative (SOBACO) Spring Research Symposium
Minneapolis, MN. May 14, 2015

WikiBrain: Making Computer Programs Smarter with Knowledge from Wikipedia
Invited Demo at The Social Media and Business Analytics Collaborative (SOBACO) Spring Research Symposium
Minneapolis, MN. May 6, 2014