Penquiry
A Pen-based Interactive In-situ Q&A System Leveraging LLMs
Jeongmin Rhee, Changhee Lee, Hyunwoo Kim, Kiroong Choe, Bohyoung Kim, Sungahn Ko, and Jinwook Seo / 2026
Penquiry
Participants
  • Jeongmin Rhee, Seoul National University
  • Changhee Lee, Pohang University of Science and Technology
  • Hyunwoo Kim, Seoul National University
  • Kiroong Choe, Seoul National University
  • Bohyoung Kim, Hankuk University of Foreign Studies
  • Sungahn Ko, Pohang University of Science and Technology
  • Jinwook Seo, Seoul National University
Abstract

Pen-based digital devices remain a preferred medium for active, cognitively engaging study. Concurrently, Large Language Models (LLMs) have become indispensable for self-directed learning, enabling students to clarify concepts. However, a fundamental interaction gap exists between the fluid, spatial nature of pen-based workflows and the discrete, keyboard-heavy requirements of LLMs. We present Penquiry, an in-situ question-and-answer system that bridges this gap by enabling learners to pose questions directly on digital study materials via a pen. We characterize two primary interaction challenges in this multimodal transition: a Referential Barrier, which hinders grounding fine-grained visual elements into the query context, and an Expressive Barrier, which forces learners to translate diverse, non-textual intents-such as equations and diagrams-into rigid, typed sentences. To resolve these, Penquiry introduces a mediation layer featuring Content Snapping for unambiguous referencing and Question Autocompletion to expand sparse ink keywords into rich semantic queries. Through two iterative user studies (N=16 per study), we demonstrate that Penquiry significantly reduces the cognitive and physical overhead of inquiry compared to traditional interfaces, providing a new blueprint for pen-based, in-situ AI interaction.

Related Publications
UIST '26 | Conference Full Paper
Penquiry: A Pen-based Interactive In-situ Q&A System Leveraging LLMs
Jeongmin Rhee, Changhee Lee, Hyunwoo Kim, Kiroong Choe, Bohyoung Kim, Sungahn Ko, and Jinwook Seo