Research

Multimodal Language Models

Integrating vision, speech, and text into unified language models capable of grounded understanding and generation.

Language models now take in images and audio as well as text. We study how to combine vision, speech, and language in one model so that its reasoning is grounded in what it sees and hears. Our work covers representation learning across modalities, retrieval-augmented multimodal dialogue, and the cases where current multimodal LLMs fail to reason over fine-grained visual or acoustic evidence.

Current directions include multimodal dialogue response retrieval, vision-grounded conversation, and models that reason over text and non-text signals together without one modality dominating the other.

Related papers

Social Intelligence of AI Agents

Equipping dialogue agents with empathy, pragmatics, and context-aware behavior for long-term companionship.

We study how dialogue agents can track context over long conversations, decide when to respond as well as what to say, reason about what the other person believes and feels, and adapt to cultural and interpersonal norms.

Current work includes timely response generation, empathetic and proactive dialogue, long-term memory in conversational agents, and the evaluation of conversational understanding across languages.

Related papers

Evaluation of LLMs

Building benchmarks and evaluation methods that measure what matters when models are deployed.

Benchmarks shape which models get built. We design benchmarks and evaluation methods for the capabilities that matter in practice, even when they are hard to score: conversational understanding, multilingual competence, the effect of tokenization choices, and task-grounded reasoning.

Past and ongoing work includes Korean dialogue and language understanding benchmarks, empirical studies of tokenization strategies, and tests of how frontier models fail on culturally specific tasks.

Related papers