Publications

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components
Toby Jia-Jun Li, Lindsay Popowski, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [GitHub]

Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M. Mitchell, and Brad A. Myers
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
🏆 Best Paper Award
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications
Ritam Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang ‘Anthony’ Chen
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2020)
[Paper PDF] [ACM DL] [Demo Video]

Privacy-Preserving Script Sharing in GUI-based Programming-by-Demonstration Systems
Toby Jia-Jun Li, Jingya Chen, Brandon Canfield, and Brad A. Myers
Proceedings of the ACM on Human-Computer Interaction (CSCW 2020)
[Paper PDF] [ACM DL]

Interactive Task Learning from GUI-Grounded Natural Language Instructions and Demonstrations
Toby Jia-Jun Li, Tom Mitchell, and Brad A. Myers
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020): System Demonstrations
[Paper PDF] [ACL Anthology] [Talk Video]

PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations
Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST 2019)
[Paper PDF] [ACM DL] [Demo Video] [Talk Video]

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers
Proceedings of the 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2018)
[Paper PDF] [IEEE Xplore] [Demo Video]

KITE: Building conversational bots from mobile apps
Toby Jia-Jun Li and Oriana Riva
Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018)
[Paper PDF] [ACM DL] [Talk Video]

PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps
Yuanchun Li, Fanglin Chen, Toby Jia-jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal, and Jason I. Hong
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2017)
[Paper PDF] [ACM DL] [Website]

Programming IoT Devices by Demonstration Using Mobile Apps
Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A. Myers
Proceedings of the International Symposium on End User Development (IS-EUD 2017)
🏆 Best Paper Award
[Paper PDF] [SpringerLink]

SUGILITE: Creating Multimodal Smartphone Automation by Demonstration
Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017)
🏅 Best Paper Honorable Mention Award
[Paper PDF] [ACM DL] [Demo Video] [GitHub]

Not at Home on the Range: Peer Production and the Urban/Rural Divide
Isaac Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning, and Brent Hecht
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016)
[Paper PDF] [ACM DL]

Leveraging Advances in Natural Language Processing to Better Understand Tobler’s First Law of Geography
Toby Jia-Jun Li, Shilad Sen, and Brent Hecht
Proceedings of the ACM Conference on Advances in Geographic Information Systems (SIGSPATIAL 2014)
[Paper PDF] [ACM DL]

WikiBrain: Democratizing Computation on Wikipedia
Shilad Sen, Toby Jia-Jun Li, WikiBrain Team, and Brent Hecht
Proceedings of the 10th International Symposium on Open Collaboration (OpenSym / WikiSym 2014)
[Paper PDF] [ACM DL]