Lu Dong (董璐)

Ph.D. Student

Department of Computer Science and Engineering
University at Buffalo, SUNY (UB)
301 Davis Hall, Buffalo, New York, US, 14260

Email: ludong@buffalo.edu

About Me

I'm a Ph.D. candidate in the Department of Computer Science and Engineering (CSE), University at Buffalo-The State University of New York (UB) , working with Prof. Ifeoma Nwogu at the Human Behavior Modeling Lab. Prior to my Ph.D. studies, I obtained a Master's degree in Computer Science and Technology from the School of Computer Science and Technology at Xi'an Jiaotong University (XJTU) in China, where I was advised by Prof. Xinyu Yang at the YLab .

My research focuses on Human-Centered Physical AI. My work spans Embodied AI, Multimodal Foundation Models, Generative AI, 3D Human Behavior Modeling, Video Understanding, and Human-Robot Interaction. I develop AI systems that understand, generate, and interact with human behaviors in the physical world. I am currently on the job market for full-time Research Scientist, Research Engineer, and Applied Scientist positions in academia and industry.

News []

2026.08: Secured $10,000 in industry funding to support research at the University at Buffalo Human Behavior Modeling Lab (HBML).
2026.08: Our paper, MistyPilot, has been accepted to the 14th ACVR (Assistive Computer Vision and Robotics) Workshop at ECCV 2026 as an Oral & Poster presentation.
2026.05: Started a summer Research Internship at MaxInsights, a startup building the data foundation for Physical AI.
2026.03: Two papers in AI for Education, “ConfusionBench” and “InterventionLens,” were accepted to the CVPR 2026 CV4Edu - Computer Vision for Education Workshop.
2025.12: I’m honored to share that I received the 2025 CSE “PhD Research Award”. I am deeply grateful to my advisor for her continuous guidance and encouragement, and to our lab and all my collaborators for their unwavering support.
2024.09: One paper on Agentic Frameworks for Human and Social Robot Interaction has been accepted by IROS 2025 .
2025.05: Started my summer Research Internship journey at NEC Laboratories America, Inc. in Princeton, NJ. (on-site)
2025.04: I successfully passed the Ph.D. dissertation proposal defense at CSE Department of UB.
2025.03: I was invited to give a talk on AI for Human Behavior at Women in Tech Western New York .
2024.12: I’m honored to share that I have been awarded the Best AI Project Award at the 2024 CSE Poster Competition.
2024.11: I will be presenting my work at the EMNLP 2024 Conference on November 12. See you in Miami!
2024.10: After a competitive process, five outstanding candidates have advanced to the final phase of Research Acceleration . Congratulations to them and best wishes for their continued success!
2024.10: We are launching an initiative called the Research Acceleration Program , aimed at helping early-stage AI researchers accelerate their growth, focusing on AIGC, Human Motion Generation, with applications in children's educational interactions, sign language synthesis, and social intelligence. UB master's and undergraduate students are encouraged to apply. If you're interested, please feel free to send me your resume.
2024.10: Happy to be invited as a speaker for a panel discussion on 'AI Research and Career Development' at UB.
2024.09: I’ll be presenting my work at ECCV 2024 on September 30 and October 4. See you in Milan, Italy!
2024.09: One first-author paper on 3D sign language motion generation has been accepted by EMNLP 2024 .
2024.09: I’m honored to share that I have received the DEI Leadership Award at the IJCB 2024 Conference.
2024.09: I’m excited to serve as the Local Student Chair for the IJCB 2024 Conference Organizing Team.
2024.08: Two papers have been accepted by ECCV 2024: one on enhancing facial expression inference, and the other on open-domain multi-person motion generation.
2024.08: I am honored to share that my work has been recognized by National AI Institute for Exceptional Education (AI4ExceptionalEd), the Institute for Artificial Intelligence and Data Science (IAD) and UB-CSE. [Read] [More].
2024.05: I will present my work at FG 2024 Conference on May 27 and 30. See you in Istanbul, Turkey.
2024.04: Two papers on sign language generation have been accepted by FG 2024.
2024.03: I participated in the AI4EE: Year 1 Annual Review Site Visit and presented our work in the event.
2023.09: The National AI Institute for Exceptional Education (AI4EE) will support my research aimed at analyzing and modeling children's non-verbal behaviors, such as facial expressions and body gestures, to enhance educational performance.
2023.08: I was invited to give a talk on Internships in the course CSE501, Fall2023, UB.
2023.07: One paper on human motion synthesis has been accepted by ACM MM 2023.
2023.06: I successfully passed my Oral Qualifying Exam (OQE) and became a PhD candidate at UB.
2023.05: Started my research intern journey at InnoPeak Technology (OPPO US Research) in Seattle, WA. (on-site)
2022.11: I was invited to be a judge of the 2022 UB Hacking Competition!
2022.10: I was invited to give a talk in the course CSE501, Fall2022, UB.
2022.05: Started my research intern journey at InnoPeak Technology (OPPO US Research) in Palo Alto, CA. (on-site)
2021.08: Started my Ph.D. journey following my advisor at UB, Buffalo, NY.

Selected Research

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration
Lu Dong*, Xiao Wang*, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju
European Conference on Computer Vision (ECCV 2026) & ACVR
Code PDF Project Webpage

ConfusionBench: An Expert-Validated Benchmark for Confusion Recognition and Localization in Educational Videos
Lu Dong*, Xiao Wang*, Mark G Frank, Srirangaraj Setlur, Venu Govindaraju, Ifeoma Nwogu.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (ECCV 2026) & CV4Edu.
DOI PDF BibTeX Project Webpage

InterventionLens: A Multi-Agent Framework for Detecting ASD Intervention Strategies in Parent-Child Shared Reading
Lu Dong*, Xiao Wang*, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju.
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (ECCV 2026) & CV4Edu.
DOI PDF BibTeX Project Webpage

AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot
Lu Dong*, Xiao Wang*, Sahana Rangasrinivasan, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju. (*Equal contribution)
International Conference on Intelligent Robots and Systems [IROS2025]
DOI PDF BibTeX Project Webpage

Word-Conditioned 3D American Sign Language Motion Generation
Lu Dong, Xiao Wang, Ifeoma Nwogu.
In Findings of the Association for Computational Linguistics [EMNLP 2024]
DOI PDF BibTeX Project Webpage

Ig3D: Integrating 3D Face Representations in Facial Expression Inference
Lu Dong*, Xiao Wang*, Srirangaraj Setlur, Venu Govindaraju, Ifeoma Nwogu.
The 18th European Conference on Computer Vision [ECCV 2024] Workshop.
DOI PDF BibTeX Project Webpage

SignAvatar: Sign Language 3D Motion Reconstruction and Generation
Lu Dong, Lipisha Chaudhary, Fei Xu, Xiao Wang, Mason Lary, Ifeoma Nwogu.
The 18th IEEE International Conference on Automatic Face and Gesture Recognition [FG 2024].
DOI PDF BibTeX Project Webpage

A Large-Scale 3D Representation Dataset and Benchmark for Continuous Sign Language Understanding
Lipisha Chaudhary, Enjamamul Hoq, Lu Dong, Henry Adler, Ifeoma Nwogu.
The 18th IEEE International Conference on Automatic Face and Gesture Recognition [FG 2026].
DOI PDF BibTeX Project Webpage

Towards Open Domain Text-Driven Synthesis of Multi-Person Motions
Mengyi Shan, Lu Dong, Yutao Han, Yuan Yao, Tao Liu, Ifeoma Nwogu, Guo-Jun Qi, Mitch Hill.
The 18th European Conference on Computer Vision [ECCV 2024]
DOI PDF BibTeX Project Webpage

A Comparative Study of Video-based Human Representations for American Sign Language Alphabet Generation
FeiXu, Lipisha Chaudhary, Lu Dong, Srirangaraj Setlur, Venu Govindaraju, Ifeoma Nwogu.
The 18th IEEE International Conference on Automatic Face and Gesture Recognition [FG 2024].
DOI PDF BibTeX

Language-guided Human Motion Synthesis with Atomic Actions
Yuanhao Zhai, Mingzhen Huang, Tianyu Luan, Lu Dong , Ifeoma Nwogu, Siwei Lyu, David Doermann, Junsong Yuan
ACM Multimedia [MM 2023].
Please check title link for more details.
DOI PDF BibTeX Project Webpage

Exploring the General Melodic Characteristics of XinTianYou Folk Songs
Juan Li, Lu Dong, Jianhang Ding, Xinyu Yang
12th Sound and Music Computing Conference(SMC)
DOI PDF BibTeX Project Webpage

Selected Systems & Prototypes

A smooth 3D Human Pose Estimation pipeline for video in the wild
The majority of current models have been trained on limited close set data, which often results in subpar performance when applied to real-world video data. One of the major challenges in this context is the issue of self-occlusion, which refers to instances where parts of a subject's body obstruct or cover other parts. This results in a less smooth and precise performance. This project aims to tackle this issue.
Please check title link for more details.

A Training ChatBot to facilitate medical knowledge dissemination in underdeveloped regions
This project is undertaken in providing for support for a non-profit organization with the aim of addressing the lack of medical knowledge in underdeveloped regions of India. This shortfall leads to an alarming number of preventable fatalities each year. We have developed a training chatbot to enhance the understanding of medical knowledge and local resources by volunteers and the local population. The chatbot will serve as an educational tool to improve medical literacy and ultimately, save lives.
Please check title link for more details.

A Covid Analysis Search Engine: Dissecting Twitter data to analyze government & public attitude towards Covid and Vaccines
Do authoritative figures have any influence on the trend of COVID-19, what is the attitude of the general public toward COVID-19 and vaccines, whether changes in public attitudes are influenced by authoritative speech? The aim of this project is to design and develop a full-stack search engine that integrates advanced functionalities for retrieval, translation, sentiment analysis, attitude tracking, topic & area statistics, and disinformation detection from multiple dimensions.
Please check title link for more details.

Expert-Validated Confusion Benchmark Demo

ConfusionBench Project
Confusion is a critical signal in e-learning, yet high-quality benchmarks for confusion understanding and localization remain limited. Existing datasets often lack expert validation, contain noisy and inconsistent labels, and rely on coarse temporal annotations that fail to capture brief confusion events. To address these limitations, ConfuBench provides an expert-validated confusion benchmark with Golden Clips and Long-Video Localization Annotations.
Please check title link for more details.

Mentorship Program

Research Acceleration Program
Building on my own research areas, I initiated the Research Acceleration Program to support the rapid growth of early-stage AI researchers, with a focus on human-centered multimodal intelligence—spanning generative AI, multimodal large models, and LLM-based agentic systems for applications in education, sign language communication, and digital humans. The program is designed to mentor students through the journey of turning research ideas into concrete, deployable pipeline systems.
Please check title link for more details.

My Papper Digest

LatestPaper Digest
This is an ongoing personal research digest for learning, synthesis, and discussion. I collect and summarize recent papers on topics I’m exploring, aiming to keep it a simple space for reading, discussion, and idea sharing. Current topics include generative models (GenAI), chain-of-thought (CoT), mixture-of-experts (MoE), motion generation, dyadic and multi-agent interaction, and affective computing / behavior analysis. I curate papers that offer a clear one-sentence problem statement, concise contributions, useful methods, high-level insights, and key implementation logic across these areas. I hope this digest helps identify and capture research trends, spark ideas discussion, and support collaboration.

Experience

Research Assistant

Research Assistant, National AI Institute for Exceptional Education, University at Buffalo-SUNY (UB), Sep 2023 – Present
Research Assistant, Human Behavior Modeling Lab, University at Buffalo-SUNY (UB), Aug 2021 – Present
Research Assistant, Rochester Institute of Technology (RIT), Aug 2020 – May 2021
Research Assistant, Xi’an Jiaotong University (XJTU), Aug 2013 – May 2016

Research Internship

Research Intern, MaxInsights, Santa Clara, CA, On-Site, May 2026 – Aug 2026
Research Intern, NEC Laboratories America, Princeton, NJ, On-Site, May 2025 – Aug 2025
Research Intern, InnoPeak Technology, Seattle, WA, On-Site, June 2023 – Aug 2023
Research Intern, InnoPeak Technology, Palo Alto, CA, On-Site, May 2022 – Aug 2022

Selected Awards & Honors

PhD Research Award , UB, 2025.
Best AI Project , UB 2024.
IJCB Conference Leadership Award , 2024.
ECCV Conference Travel Grant , 2024.
National Graduate Academic Scholarship , 2013-2016.
National Endeavor Undergraduate Scholarship for Outstanding Students , 2010-2011
Excellent Graduate Student Honor, 2014-2016 ; Excellent Undergraduate Student Honor, 2010-2011.

Academic Services

Conference Organization

IEEE International Joint Conference on Biometrics (IJCB 2024): Served as Local Student Chair, working with Prof. Ifeoma Nwogu (Local Arrangements Chair) and Prof. Nalini Ratha (General Chair).

Conference & Journal Reviewer

Association for Computational Linguistics (ACL Rolling Review ARR), February & July 2025
ACM International Conference on Multimedia (ACM MM), 2022, 2023, 2024
Computer Vision and Image Understanding (CVIU), 2025, 2026
Machine Vision and Applications (Springer Nature), 2024, 2025
IEEE Transactions on Affective Computing (IEEE TAFFC), 2024

Invited Talks & Competition Judge

2025.03: I was invited to give a talk on AI for Human Behavior at Women in Tech Western New York.
2024.10: I was invited as a speaker for a panel discussion on 'AI Research and Career Development' at UB.
2023.8: I was invited to give a talk on Internships in the course CSE501, Fall2023, UB.
2022.11.06 Invited as a Judge for 2022 UB Hacking Competition.
2022.10: Delivering a talk, introducing Human Pose Estimation and Ph.D Life Guidance in the course CSE-501, Fall2022.