SUGILITE – Programing by Demonstration for Mobile Intelligent Personal Assistants
SUGILITE is a programming-by-demonstration (PBD) system that enables users to program intelligent personal assistants on smartphones. When the user gives verbal commands that the assistant does not know how to execute, the user can demonstrate the task using SUGILITE by directly manipulating arbitrary third-party apps’ user interface. By leveraging the verbal instructions, the demonstrated procedures, and the apps’ UI hierarchy structures, SUGILITE can automatically generalize the script from the recorded actions, so SUGILITE learns how to perform tasks with different variations and parameters from a single demonstration. The follow-up system EPIDOSITE also extends SUGILITE to support programming for smart home devices. Read our CHI ’17 Paper about SUGILITE / Watch a SUGILITE demo video
Atlasify – Spatialization, Visualization and Spatial Information Retrival
Atlasify is a novel information retrieval / interactive visualization system supporting exploratory search. As the Lead Student Researcher and Head Developer, I re-implemented the system with Leaflet on the front end and WikiBrain on the back end, enabling the system to dynamically compute the semantic relatedness for any given keywords and render the interactive map instantly. Since its beta release in June 2015, Atlasify has acquired thousands of active users and been featured on Wired, Phys.org and ACM Newsletter. Based on Atlasify, we are now conducting a variety of user behavior studies and investigating the HCI aspect of spatialization and spatial information retrieval system design. Read more about Atlasify or Try out the beta version of Atlasify
WikiBrain – A Java Library for Wikipedia-based Algorithms
WikiBrain is a Java-based library/framework wrote by Shilad Sen, Toby J Li, Brent Hecht and a group of undergraduate students at Macalester College. It democratizes access to a range of Wikipedia-based algorithms and technologies, enabling anyone with basic Java programming skills to utilize state-of-art semantic relatedness algorithms, page view data analysis and spatial queries in a few lines of code and easily analyze terabytes of Wikipedia data. After its debut at OpenSym/WikiSym ’14, WikiBrain has been used by many in both academia and industry. Learn more about WikiBrain / WikiBrain homepage
Peer-reviewed Conference Papers
[C.1] Toby Jia-Jun Li and Oriana Riva. 2018. KITE: Building conversational bots from mobile apps. Proceedings of the 16th ACM International Conference on Mobile Systems, Applications, and Services (MobiSys 2018).
[C.2] Yuanchun Li, Fanglin Chen, Toby Jia-jun Li, Yao Guo, Gang Huang, Matthew Fredrikson, Yuvraj Agarwal and Jason I. Hong. PrivacyStreams: Enabling Transparency in Personal Data Processing for Mobile Apps. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (PACM IMWUT / UbiComp 2017) [Paper PDF][ACM DL][Website]
[C.3] Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen and Brad A. Myers. 2017. Programming IoT Devices by Demonstration Using Mobile Apps. Proceedings of the International Symposium on End User Development (IS-EUD 2017). Best Paper Award. [Paper PDF][SpringerLink]
[C.4] Toby Jia-Jun Li, Amos Azaria and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017). Best Paper Honorable Mention Award. [Paper PDF][ACM DL][Video DEMO]
[C.5] Isaac Johnson, Yilun Lin, Toby Jia-Jun Li, Andrew Hall, Aaron Halfaker, Johannes Schöning and Brent Hecht. 2016. Not at Home on the Range: Peer Production and the Urban/Rural Divide. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016). [Paper PDF][ACM DL]
[C.6] Toby Jia-Jun Li, Shilad Sen and Brent Hecht. 2014. Leveraging Advances in Natural Language Processing to Better Understand Tobler’s First Law of Geography. Proceedings of the 22nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL 2014). [Paper PDF][ACM DL]
[C.7] Shilad Sen, Toby Jia-Jun Li, WikiBrain Team and Brent Hecht. 2014. WikiBrain: Democratizing Computation on Wikipedia. Proceedings of the 10th International Symposium on Open Collaboration (OpenSym / WikiSym 2014). [Paper PDF][ACM DL]
[B.1] Toby Jia-Jun Li, Igor Labutov, Brad A. Myers, Amos Azaria, Alexander I. Rudnicky and Tom M. Mitchell. An End User Development Approach for Failure Handling in Goal-oriented Conversational Agents. Chapter of Studies in Conversational UX Design, Robert J. Moore, Margaret H. Szymanski, Raphael Arar, Guang-Jie Ren eds. Springer. In press.
[B.2] Brad A. Myers, Andrew Ko, Chris Scaffidi, Stephen Oney, YoungSeok Yoon, Kerry Chang, Mary Beth Kery and Toby Jia-Jun Li. Making End User Development More Natural. Chapter of New Perspectives in End-User Development, Fabio Paternò and Volker Wulf, eds. Springer, 2017. [SpringerLink]
[WkSp.1] Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Tom M. Mitchell and Brad A. Myers. 2018. Supporting Co-adaptive Human-Agent Relationship through Programming by Demonstration using Existing GUIs. Rethinking Interaction CHI 2018 Workshop. [Paper PDF]
[WkSp.2] Toby Jia-Jun Li. 2017. End User Mobile Task Automation using Multimodal Programming by Demonstration. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2017) Graduate Consortium. [Paper PDF][IEEE Xplore]
[WkSp.3] Toby Jia-Jun Li, Brad A. Myers, Amos Azaria, Igor Labutov, Alexander I. Rudnicky and Tom M. Mitchell. 2017. Designing a Conversational Interface for a Multimodal Smartphone Programming-by-Demonstration Agent. Conversational UX Design CHI 2017 Workshop. [Paper PDF]
[WkSp.4] Toby Jia-Jun Li and Brad A. Myers. 2016. Smartphone Text Entry in Cross-Application Tasks. CHI 2016 Workshop on Inviscid Text Entry and Beyond. [Paper PDF]
Invited Talks and Presentations
[P.1] Toby Jia-Jun Li, Josh Ford, Doug Downey, Brent Hecht, Vijay Murganoor and Shilad Sen. 2015. Atlasify – The Geography of Everything. 3M Science and Engineering Symposium. St Paul, MN. June 25, 2015.
[P.2] Toby Jia-Jun Li, Josh Ford, Doug Downey, Brent Hecht, Vijay Murganoor and Shilad Sen. 2015. Atlasify – The Geography of Everything. The Social Media and Business Analytics Collaborative (SOBACO) Spring Research Symposium. Minneapolis, MN. May 14, 2015.
[P.3] Toby Jia-Jun Li and Brent Hecht. 2014. WikiBrain: Making Computer Programs Smarter with Knowledge from Wikipedia. The Social Media and Business Analytics Collaborative (SOBACO) Spring Research Symposium. Minneapolis, MN. May 6, 2014.