个人简介
博士，教授，硕士生导师，现为陕西师范大学教育学院教育技术系教师。获法国中央理工计算机科学博士学位，目前从事智能交互、脑机交互及虚拟现实在教育中的应用等研究，著有专著1部，已有30余篇成果发表在领域内有影响力的国际会议及SCI期刊上，多篇SCI期刊审稿人，人工智能学会CAAI智能交互专委会委员，主持国家自然科学基金2项、陕西省自然科学基金1项等，参与多个项目。
教育经历
北京师范大学 获双学位学士（主修教育技术学，辅修英语）
北京交通大学 获教育技术学理学硕士
法国里昂中央理工学校 获计算机科学博士学位
课程
Python程序语言设计
人机交互与在线学习设计原则
教育数据挖掘
数据分析与机器学习
虚拟现实教学资源建设与应用
教育软件与大数据
研究领域
教育技术 人机交互
主要研究方向：虚拟现实、3D学习资源、智能交互在教育中应用研究、认知情绪识别与教育脑机接口研究、人工智能技术精准帮扶农村教师的难点与解决路径研究
学术兼职
人工智能学会CAAI智能交互专委会委员
科研项目
1. 国家自然科学基金面上项目，智能教学系统中基于脑机接口的困惑情绪识别与双向调节策略研究
2. 国家自然科学基金青年项目，在线学习自适应交互中认知负荷与学习情绪的脑电表征及认知状态的评估与调整
3. 陕西省自然科学基金，在线学习中基于计算机视觉的注意力识别与智能交互研究
4. 陕西师范大学中央高校基本科研基金，基于眼动和表情识别的MOOCs视频视觉注意力机制研究
1. 科技部国家重点研发计划“社会治理与智慧社会科技支撑(平安中国)”专项，农村地区教师教学能力智能评测与教学精准辅助技术研究
2. 横向，基于EEG的人脑焦虑、抑郁情绪量化评价方法参与
3. 陕西省自然科学基金，基于多分辨率特征的无线传感器网络能量最优化模型及求解算法研究
4. 横向，基于家居智能看护机器人的老人反常行为感知关键技术研究
5. 横向，教师教育信息化能力研修平台资源建设及应用
6. 参与法国“eGonomy”项⽬，为法国9个知名企业、研究所及博物馆合作项目，包括法国国家博物馆联合会(RMN)，该项目旨在通过用户行为分析来为大型图片数据库交互提供多模、自然、智能的交互。主要负责与实施用户活动追踪分析与探索模块。
7. 参与德国电信与美国Mozilla基金会关于Firefox OS的合作项目，该项目旨在打造安全友好的Firefox智能手机系统。主要负责与实施监测与用户评估模块。
主要论文列表
基于上下文的智能感知交互模式与系统
周筠 徐韬
2015.11陕西师范大学出版社
J Bai, X Cheng, H Zhang, Y Qin, T Xu, Y Zhou.
Can AI-generated pedagogical agents (AIPA) replace human teacher in picture book videos? The effects of appearance and voice of AIPA on children’s learning.
Education and Information Technologies, 1-21,2025,SSCI一区,IF：4.8
(Abstract)
Objective. This study examines the application of AI agents in education through a systematic literature review. Empirical studies published after November 2023 were retrieved and screened from four databases including Web of Science, ERIC, IEEE Xplore, and the ACM Digital Library. Eleven studies were finally retrieved for qualitative analysis. The results show that, as adaptive teaching tools, AI agents can improve learners' academic performance, knowledge transfer, social interaction
quality, and emotional experience. Their educational applications span all academic stages and present a spiral pattern of progression. They focus on stimulating interest and conducting basic inquiry in the primary school stage, cultivating complex cognitive skills in the middle school stage, and training advanced professional abilities in the university stage. In terms of design and implementation, AI agents are predominantly centered on conversational interaction and embedded within
existing instructional structures. Their functional design is informed by learning theories such as scaffolding and constructivism. These findings synthesize empirical evidence to illuminate the progressive application of AI agents in education.
Y Zhao, H Sun, Z Fu, H Kou, Y Zhang, Y Zhou
Applications of AI agents in Education: A Systematic Review Based on Empirical Studies.
2026 8th International Conference on Computer Science and Technologies in Education (CSTE), 1568 1572, 2026, EI Conference.
(Abstract)
Objective.Avatars are users’ digital representations in a virtual reality (VR) environment. As a core medium in VR, avatars may do more than merely represent users; they can actively transform learners’ perceptions, experiences, and
learning effectiveness. To explore the current research progress in this domain, this systematic review synthesizes existing studies on avatar use in educational VR, focusing on application contexts, learning effects, underlying mechanisms, theoretical foundations, and design considerations. Data were sourced from three
databases: Web of Science, IEEE Xplore, and Google Scholar. Following a rigorous screening process, 15 articles were included for analysis. The results indicate that avatars have been widely adopted across diverse educational contexts, including teacher training, language learning and medical education. They influence learning outcomes through three core mechanisms: cognitive, affective-motivational, and social interaction pathways. The reviewed studies are predominantly grounded in theories including embodied cognition, social presence, the illusion of body ownership, and the Proteus effect. Moreover, existing research particularly focuses on avatar visual presentation, interactive functionality, and socio-psychological aspects. This review provides a structured synthesis of existing evidence and offers a systematic referenc
J Wang, Y Zhang, J Shu, Y Zhou, Y Zhou
Avatars in Educational Virtual Reality: A Systematic Review.
2026 8th International Conference on Computer Science and Technologies in Education (CSTE), 1591 1596, 2026, EI Conference.
(Abstract)
Objective.Immersive Virtual Reality (IVR) provides unique potential for facilitating STEM learning through realistic and interactive visualization of complex concepts. However, its effectiveness may be impacted by individual differences in visuospatial imagery (VSI), referring to learners’ tendency or ability to visualize and mentally manipulate spatial information. Visual fidelity in IVR can serve as either a scaffold for mental imagery or a source of additional cognitive load, and its interaction with VSI remains unclear. To address this gap, we examined the effects of learning material fidelity (high vs. low) and self-perceived VSI (high vs. low) on learning fungal biology in IVR. This study adopted a 2 × 2 between-group design involving 120 college students (61 females, 59 males), and measured learning performance, cognitive load, motivation, learning experience, and emotions. Results revealed that VSI significantly moderated the effects of fidelity in IVR. Learners with low VSI benefited from high-fidelity learning materials, which compensated for their spatial processing limitations and improved retention, perceived ease of use, and positive emotions. In contrast, learners with high VSI performed similarly under both high- and low-fidelity conditions; however, they reported lower extraneous cognitive load, reduced fear of failure and negative emotions, and higher perceived usefulness, perceived ease of use, and satisfaction when using low-fidelity materials. These results extend the abilities-as-compensator hypothesis to IVR and self-reported VSI contexts and highlight the importance of adapting visual fidelity to learners’ VSI when designing visualizations for STEM education.
Bingyu Dong, Yun Zhou, Yueyao Qu, Junjun Zhai, Shu Zhao, Jialin Yu, Yihang Qin & Tao Xu
Burden or benefit? Effects of visuospatial imagery and fidelity on learning in immersive virtual reality
Virtual Reality,2026,SCIE一区,IF:5.4
(Abstract)
Objective.Multi-User Virtual Environments (MUVEs) have gained increasing attention in collaborative learning research due to their interactive affordances. However, most current MUVEs remain desktop-based, limiting their ability to simulate authentic learning contexts. Immersive virtual reality (IVR), delivered through head-mounted displays that engage learners' peripheral vision, can enhance their sense of presence and immersion within virtual environments. To explore this potential, the present study proposes a multidimensional design framework that encompasses social, motivational, cognitive, and experiential dimensions and uses it to develop an Immersive Multi-User Virtual Environment (MUIVE). The effectiveness of MUIVE was evaluated using measures of learning performance, social presence and collaboration quality (social), learning motivation (motivational), cognitive load (cognitive), and IVR experience, trust, ease of use, and satisfaction (experiential). Results showed a higher overall performance gain for MUIVE in the gain-score analysis, although the corresponding baseline-adjusted analysis did not yield a statistically significant condition effect. MUIVE significantly enhanced social presence, learning motivation, trust, ease of use, and satisfaction, without increasing cognitive load. Based on these findings, we propose five core design principles for building effective MUIVEs: (a) face-to-face authenticity, (b) engagement enhancement, (c) extraneous cognitive load minimization, (d) avatar embodiment, and (e) system experience. Furthermore, peer familiarity moderated the effect of platform on transfer performance, with MUIVE showing a stronger advantage among unfamiliar dyads. These findings offer design implications for the development of immersive collaborative learning systems and serve as a foundation for future research on optimizing multi-user interaction and learning support in IVR-based environments.
Yueyao Qu, Yun Zhou, Jin Shu, Junjun Zhai, Jie Bai, Jialin Yu
Designing Immersive Multi-User Virtual Environments for Collaborative Learning: A Multidimensional Framework and Evaluation
IEEE Transactions on Learning Technologies,SSCI一区,IF:7.4
(Abstract)
Objective.Spatial ability is fundamental to STEM learning, as many STEM tasks require the mental representation and transformation of 2D and 3D information. Desktop virtual reality (desktop VR) has therefore been increasingly adopted for spatial training due to its interactivity and 3D representations. Textual and visual cues are widely used as instructional supports in multimedia learning, yet their distinct and combined effects on spatial training in desktop VR remain insufficiently understood. This study conducted a 2 × 2 between-subjects experiment manipulating the presence of textual explanations and spatially augmented visual cues in a desktop VR environment to examine spatial visualization and mental rotation performance. Spatial performance, learning experience, cognitive load, and visual attention allocation were measured using tests, subjective scales, and eye-tracking data. The results showed no significant main effect of textual explanations or spatially augmented visual cues on posttest performance after controlling for pretest scores, but a significant interaction between the two. The textual-explanation-only condition outperformed the combined textual-explanation-and-visual-cue condition, suggesting that the effect of textual explanations depended on whether visual cues were also present. Conditions including spatially augmented visual cues showed comparable training-phase performance to the other conditions; however, this was not accompanied by better immediate posttest performance when the cues were unavailable. Eye-tracking results further revealed that the combined cue/interactive areas received more fixations than the text areas, but this attention pattern did not translate into better posttest performance. These findings inform the application of the signaling principle within desktop VR-supported spatial training and suggest that instructional supports need to be carefully coordinated rather than simply combined.
Qian Chen, Yun Zhou, HanBing Kou, Hui Zhang, Tao Xu
Effects of textual explanations and spatially augmented visual cues on spatial visualization and mental rotation performance in desktop VR
Thinking Skills and Creativity,2027,SSCI一区,IF:5.3
(Abstract)
Objective.Instructional videos are widely used, yet their limited interactivity often leads to passive viewing. Embedding Virtual Reality (VR) as a form of generative practice, in which learners actively create content such as drawings or visual representations, offers a promising solution to foster deeper cognitive processing. This study examines the impact of integrating generative VR practice into instructional videos through an experimental design involving 100 university students. Learners engaged with vocabulary tasks under four spatial layouts that varied in the position (left vs. right) and size (small vs. large) of the interactive area, compared with a video-only control. Outcomes included retention, transfer, attention, cognitive load, and perceived enjoyment. Transfer benefits over video-only learning were observed only in the two large-area VR conditions. Among the VR conditions, right-side positioning increased attention allocation and reduced cognitive load, whereas larger interaction areas improved retention, transfer, and usability while reducing cognitive load. Eye-tracking further showed that VR-based generative practice redirected attention toward the interactive area. No significant interaction between position and size was found. These findings extend generative learning theory into VR-supported video contexts, propose the Attentional Alignment Principle and Workspace Expansion Principle, and provide design implications for interactive instructional videos.
Xiao Liu,Yun Zhou,Bingyu Dong,Hui Zhang,Lingzhi Cai,Bin Li&Tao Xu
Engage or observe? Eye-tracking and experimental evidence on integrating VR generative practice into short instructional videos
Smart Learning Environments,2026,SSCI一区,IF:17.9
(Abstract)
Objective.The rapid advancement of Artificial Intelligence Generated Content (AIGC) has revolutionized the production of instructional videos and lectures in MOOCs, opening a new chapter in their use for education. This technology enables the generation of talking instructors from text and images or clips of real people. However, the impact of behavioral fidelity and visual cues provided by these generated instructors on learning outcomes in instructional videos remains unknown. This study addresses this gap by employing a 2 × 2 within-subjects design (behavioral
fidelity: high/low; visual cues: with/without) with 36 college student participants. We assessed learning performance, social presence, satisfaction, cognitive load, humanness, and motivation, and analyzed eye movements. Results revealed that high behavioral fidelity increased social presence and more effectively engages students, leading to better transfer performance. Visual cues were found to facilitate the organization and integration of information. The generated instructors were most effective when resembling human behavior and using dynamic visual cues. However, further refinement is needed for the voice and gestures of these instructors. The findings extend the social agency theory related to animated and human instructors to the context of AIgenerated instructors. We recommend using instructors with high behavioral fidelity in instructional videos, featuring realistic beat gestures, body movements, facial expressions, and eye contact. Our study offers valuable insights into the implications of using AIGC in educational settings.
T Xu, Q Chen, Z Zhang, B Dong, H Zhang, J Bai, Y Zhou
Maximizing effectiveness of AI-generated instructors through human-like
behavior and dynamic visual cues in instructional videos: Evidence from an eye-tracking study
The Internet and Higher Education, 2025, SSCI一区, IF：10.6
(Abstract)
Objective.Agent is an adaptive system that can perceive the environment and act upon it to achieve its goals. It provides learners with efficient and convenient learning feedback support, creating vivid and authentic learning environments and experiences. Multi-agents consist of multiple autonomous agents that complete complex tasks through collaboration and communication. In recent years, the advancement of Large Language Models has driven the application of multi-agents in the field of education. This study aims to analyze the current application status of multi-agents in the field of education through a systematic literature review. Data were collected from two databases: Web of Science and IEEE Xplore. The study analyzed 18 selected articles from a qualitative perspective. The results show that multi-agents provide a significant technological path for the digital transformation of education. Their core value lies in optimizing teaching processes and performance through human-machine collaboration. However, potential risks remain, such as technical limitations and weak scenario generalization capabilities, which bring challenges to the application of multi-agents in education. These findings reveal the current application status and lay a foundation for future research.
Yuyang Zhang, Qicai Tian, Tinghui Duan, Hui Zhang, Yun Zhou
Multi-Agents in Education: A Systematic Review
2026 8th International Conference on Computer Science and Technologies in Education (CSTE), Wuhan, China, 2026, pp. 1623-1628
(Abstract)
Objective.认知失衡是一种广泛存在于学习中的不确定状态，困惑情绪是认知失衡的关键情绪表现，与知识吸收、 推理或问题解决等学习活动密切相关。研究表明，适度困惑能促进深度学习，而过度困惑则可能导致挫折感和无聊情绪，甚至使学习者放弃学习。因此，困惑情绪对学习的影响是向的。Lehman等还发现，
学习中困惑情绪的发生频率高于其他情绪。动态、精准地刻画学习者的困惑情绪，有助于全面理解其认知状态与学习特征，为教师及时调整教学策略、促进深度学习和高参与度提供依据，并为个性化学习支持与教学过程评价提供重要参考。困惑情绪的精准自动识别需要探究困惑情绪发生时的内在认知机制，并依赖于客观的测量方法。困惑情绪识别方法包括主观和客观两类。主观方法依赖自陈报告，易受主观性影响且可能干扰学习过程。客观方法包括行为识别和生理测量，行为识别通过表情、姿态、眼动和交互活动等外在表达间接测量困惑情绪；而生理测量直接捕捉身体反应，甚至测量大脑内部活动。其中，脑机接口
（Brain-Computer Interface，BCI）技术能够直接感知大脑对外界刺激的神经响应，被认为在学习情绪的生理测量和评估中具有极大潜力。Zander等将脑机接口分为主动式、反应式和被动式三类。其中，被动式脑机接口通过分析用户在任务中的脑活动，隐式识别其认知或情绪状态，在教育研究中逐渐得到重视和应用。近年来，基于脑电（Electroencephalogram，EEG）的被动式脑机接口研究在情绪识别与调节方面取得 显著进展，但现有研究多聚焦于基础情绪，对学习中典型的困惑情绪关注不足，相关教育应用仍缺乏有效技术支撑。针对这一问题，本研究采用基于脑电的脑机接口技术，构建了困惑情绪识别框架，用于识别空间任务中的困惑情绪及相关状态，比较不同方法的识别表现，分析困惑与非困惑状态下的大脑活动差异，为困惑情绪识别提供基线结果并为空间任务中困惑情绪相关神经活动特征的理解提供初步证据。
周筠，张慧，伊浩圆，徐韬，张文兰
空间任务中基于被动式脑机接口的困惑情绪识别研究
现代教育技术，2026,中文核心c刊
(Abstract)
The advancement of AI-generated content (AIGC) has made creating pedagogical agents (PAs) for multimedia learning increasingly realistic, simpler, and more efficient. However, little is known about the acceptance of AI-generated pedagogical agents (AIPAs) in picture book videos among young children aged 3–6. To address this, the present study examines the effects of AIPA appearance and voice on children’s reading performance, using a 2 × 2 between-subjects design with varied combinations of real and AI-generated voices and appearances. Children learned from one of the following picture book videos: (1) a real teacher’s voice and appearance, (2) a real teacher’s voice and AI-generated appearance, (3) an AI-synthesized voice with a real appearance, or (4) both AI-synthesized voice and appearance. The results revealed no significant difference in reading performance between the AI teacher and the real teacher. Eye-tracking data indicated that AIPA appearance and voice did not increase cognitive load, and children expressed a comparable preference for AIPAs and human teachers. These findings support the equivalence principle in children’s multimedia learning. While AIPAs may lack human micro-expressions and intonation nuances, they hold promise as complementary tools in early education.
T Xu, Y Liu, Y Jin, Y Qu, J Bai, W Zhang, Y Zhou.
From recorded to AI‐generated instructional videos: A comparison of learning performance and experience.
British Journal of Educational Technology,2024,SSCI一区,IF:6.7
(Abstract)
Generative AI (GAI) and AI- generated
content (AIGC) have been increasingly involved in
our work and daily life, providing a new learning experience
for students. This study examines whether
AI- generated instructional videos (AIIV) can facilitate
learning as effectively as traditional recorded videos
(RV). We propose an instructional video generation
pipeline that includes customized GPT (Generative
Pre- trained Transformer), text- to- speech and lip synthesis
techniques to generate videos from slides and
a clip or a photo of a human instructor. Seventy- six
students were randomly assigned to learn English
words using either AIIV or RV, with performance assessed
by retention, transfer and subjective measures
from cognitive, emotional, motivational and
social perspectives. The findings indicate that the
AIIV group performed as well as the RV group in facilitating
learning, with AIIV showing higher retention but
no significant differences in transfer. RV was found to
offer a stronger sense of social presence. Although
other subjective measures were similar between
the two groups, AIIV was perceived as slightly less
favourable. However, the AIIV was still found to be
moderately to highly attractive, addressing concerns
related to the uncanny valley effect. This research
demonstrates that AIGC can be an effective tool for
learning, offering valuable implications for the use of
GAI in educational settings.
Y Zhou, F Yi, B Dong, G Zhang, Y Zhang, T Xu.
Mental imagery scaffolding: The effects of detail richness and text load on geography learning.
Education and Information Technologies, 1-28,2024,SSCI一区,IF：4.8
(Abstract)
The growing importance of 3D animations in current teaching approaches becomes increasingly apparent, offering an effective way to visualize complex spatial concepts and processes in geography learning through outstanding visual representation and details. However, the effects of detail richness and text load of 3D animation on learning about processes remain unclear. Addressing this research gap, the present study adopts a quasi-experimental design involving four classes (n = 106) in the context of a geography lesson and evaluates four conditions in a 2 × 2 between-subjects design consisting of detail richness (high vs. low) and text load (high vs. low). The lessons on the rotation and revolution of the Earth were delivered by the same instructor across all conditions. Knowledge acquisition, cognitive load, learning experience, and emotions of students were measured. The results revealed that students were significantly better able to acquire knowledge immediately when exposed to the high detailed visuals but low text load condition. Low detail richness and high text load independently resulted in increased cognitive load. We also observed a significant effect of detail richness on the dimensions of pleasure and arousal, with higher levels of details associated with larger values in these dimensions. This research suggests that when the learning objective necessitates the engagement of mental imagery, incorporating detailed visuals can facilitate learning. The findings contribute to our understanding of how detailed imagery is linked to learning objectives about processes and expand our knowledge regarding the design of detail richness and text of 3D animation in the context of geography learning.
H Zhang, Y Zhang, T Xu, Y Zhou.
Effects of VR instructional approaches and textual cues on performance, cognitive load, and learning experience.
Educational technology research and development 72 (2), 585-607,2024,SSCI一区,IF:3.3
(Abstract)
Virtual Reality (VR) is increasingly recognized as a promising tool to enhance learning, yet research on the use of VR instructional approaches for online learning remains limited. The present study aims to address this research gap by examining the effects of VR instructional approaches and textual cues on learning. We conducted an educational VR study using a 2 × 2 + 1 between-subjects design involving 67 secondary vocational students. Participants learned computer assembly online and were exposed to either vicarious experience or direct manipulation instructional approaches, with or without textual cues. A control group received traditional online instruction using slides. We collected retention, transfer learning outcomes, cognitive load, and learning experience of students. The findings indicated that while vicarious VR had no effects on long-term retention, transfer, and learning experience, there were significant positive effects on the immediate acquisition of knowledge. Textual cues did not affect learning in general. However, for immediate knowledge gain, they did provide a positive boost to learning in VR involving direct manipulation, while they were unnecessary in vicarious VR experiences. This study contributes to how the cueing principle can be extended to educational VR contexts and expands the knowledge of vicarious VR learning.
SS Tseng, Y Zhou.
Guest editorial: Developing learner agency in smart environments.
Educational Technology & Society 27 (2), 1-3,2024,SSCI一区,IF:4.6
(Abstract)
Smart environments integrated with digital technologies have emerged as a powerful tool to support personalized, interactive, and adaptive learning experiences. The central objective of developing smart environment is to foster learner agency enabling learners to develop the ability to take ownership and manage their own learning process. However, developing learner agency presents a significant challenge. It requires researchers, practitioners, and system developers to understand the roles of teachers and students, the instructional design, and the affordances of technologies to transfer learners from passive recipients to active participants in their learning process. To address the need, this special issue collected twenty-nine articles which underwent a rigorous peer review process. The acceptance rate for this special issue was less than fifteen percent and only four papers were accepted and included. The four papers discuss the application of instructional approaches and various digital tools such as artificial intelligence, virtual reality, and mobile technologies to design smart learning environments which foster and evaluate learner agency.
Xu,T., Wang,J., Zhang,G., Zhang,L., Zhou,Y.
Confused or not: decoding brain activity and recognizing confusion in reasoning learning using EEG.
Journal of Neural Engineering,IOP Publishing,20(2),2023,SCI二区,IF：5.043
(Abstract)
Objective. Confusion is the primary epistemic emotion in the learning process, influencing
students’ engagement and whether they become frustrated or bored. However, research on
confusion in learning is still in its early stages, and there is a need to better understand how to
recognize it and what electroencephalography (EEG) signals indicate its occurrence. The present
work investigates confusion during reasoning learning using EEG, and aims to fill this gap with a
multidisciplinary approach combining educational psychology, neuroscience and computer
science. Approach. First, we design an experiment to actively and accurately induce confusion in
reasoning. Second, we propose a subjective and objective joint labeling technique to address the
label noise issue. Third, to confirm that the confused state can be distinguished from the
non-confused state, we compare and analyze the mean band power of confused and unconfused
states across five typical bands. Finally, we present an EEG database for confusion analysis, together
with benchmark results from conventional (Naive Bayes, Support Vector Machine, Random Forest,
and Artificial Neural Network) and end-to-end (Long Short Term Memory, Residual Network, and
EEGNet) machine learning methods. Main results. Findings revealed: 1. Significant differences in
the power of delta, theta, alpha, beta and lower gamma between confused and non-confused
conditions; 2. A higher attentional and cognitive load when participants were confused; and 3.
The Random Forest algorithm with time-domain features achieved a high accuracy/F1 score
(88.06%/0.88 for the subject-dependent approach and 84.43%/0.84 for the subject-independent
approach) in the binary classification of the confused and non-confused states. Significance. The
study advances our understanding of confusion and provides practical insights for recognizing and
analyzing it in the learning process. It extends existing theories on the differences between confused
and non-confused states during learning and contributes to the cognitive-affective model. The
research enables researchers, educators, and practitioners to monitor confusion, develop adaptive
systems, and test recognition approaches.
Xu,T., Dang,W., Wang,J.,Zhou,Y.
DAGAM: a domain adversarial graph attention model for subject-independent EEG-based emotion recognition.
Journal of Neural Engineering,IOP Publishing,20(1),2023,SCI二区,IF：5.043
(Abstract)
Objective. Due to individual differences in electroencephalogram (EEG) signals, the learning model
built by the subject-dependent technique from one person’s data would be inaccurate when applied
to another person for emotion recognition. Thus, the subject-dependent approach for emotion
recognition may result in poor generalization performance when compared to the
subject-independent approach. However, existing studies have attempted but have not fully utilized
EEG’s topology, nor have they solved the problem caused by the difference in data distribution
between the source and target domains. Approach. To eliminate individual differences in EEG
signals, this paper proposes the domain adversarial graph attention model, a novel EEG-based
emotion recognition model. The basic idea is to generate a graph using biological topology to
model multichannel EEG signals. Graph theory can topologically describe and analyze EEG
channel relationships and mutual dependencies. Then, unlike other graph convolutional networks,
self-attention pooling is used to benefit from the extraction of salient EEG features from the graph,
effectively improving performance. Finally, following graph pooling, the domain adversarial model
based on the graph is used to identify and handle EEG variation across subjects, achieving good
generalizability efficiently. Main Results. We conduct extensive evaluations on two benchmark data
sets (SEED and SEED IV) and obtain cutting-edge results in subject-independent emotion
recognition. Our model boosts the SEED accuracy to 92.59% (4.06% improvement) with the
lowest standard deviation (STD) of 3.21% (2.46% decrements) and SEED IV accuracy to 80.74%
(6.90% improvement) with the lowest STD of 4.14% (3.88% decrements), respectively. The
computational complexity is drastically reduced in comparison to similar efforts (33 times lower).
Significance. We have developed a model that significantly reduces the computation time while
maintaining accuracy, making EEG-based emotion decoding more practical and generalizable.
Zhou, Y., Xu, T., Yang, H., Li, S.
Improving Spatial Visualization and Mental Rotation Using FORSpatial Through Shapes and Letters in Virtual Environment.
IEEE Transactions on Learning Technologies,IEEE Education Society, 15(3):326-337,2022,SSCI一区,IF：4.433
(Abstract)
Existing research on spatial ability recognizes the critical role played by spatial visualization and mental rotation. Recent evidence suggests that external visualization and manipulation can boost spatial thinking.
The virtual environment provides an exciting opportunity so that many spatial ability training tasks based on reading printed illustrations can be migrated to a highly 3-D interactive and visualized environment. However, few studies have employed virtual reality (VR)
technology to improve spatial visualization and mental rotation. In addition, the design of training contents and corresponding VR applications are still lacking. In this work, we propose FORSpatial, a system mainly for spatial ability training in a virtual environment. First, in this article, we design a novel scheme and principles for generating tasks, involving spatial visualization
and mental rotation through flexible combinations of shapes and letters. Based on this, we create testing questions and a FORSpatial training application in desktop VR. FORSpatial and its components are made publicly available and free to use. To evaluate the performance of spatial training, verify the usability of the FORSpatial application, and analyze learning behavior, we organized a user study with 49 .participants, including an experimental group and a control group. The comparison between experimental and control groups shows the significant improvement of spatial skills through training. The analysis of interaction logging data and subjective comments reveals how FORSpatial supports spatial thinking.
Pi, Z., Deng, L., Wang, X., Guo, P., Xu, T., Zhou, Y.
The influences of a virtual instructor's voice and appearance on learning from video lectures.
Journal of Computer Assisted Learning,Wiley, 38(6):1703-1713,2022,SSCI一区,IF：3.761
(Abstract)
Background: Video lectures which include the instructor's presence are becoming
increasingly popular. Presenting a real human does, however, entail higher financial
and time costs in making videos, and one innovative approach to reduce costs has
been to generate a virtual speaking instructor.
Objectives: The current study examined whether the use of a virtual instructor in
video lectures would facilitate learning as well as a human instructor, and whether
manipulating the virtual instructor's characteristics (i.e., voice and appearance) might
optimize the effectiveness of the virtual instructor.
Methods: Our study set four conditions. In the control condition, students watched a
human instructor. In the experiment conditions, students watched one of (a) a virtual
instructor which used the human instructor's voice and an AI image, (b) a virtual
instructor which spoke in an AI voice with an AI image made to speak using text-to-speech and lip synthesis techniques, or (c) a virtual instructor with used an AI voice
and an AI likable-image of an instructor.
Results and Conclusions: The AI likable instructor condition had a significant positive
effect on students' learning performance and motivation, without decreasing the
attention students paid to the learning materials.
Implications: Our findings suggest that instructional video designers can make use of
AI voices and AI images of likable humans as instructors to motivate students and
enhance their learning performance.
Xu, T., Zhou, Y., Hou, Z., Zhang, W.
Decode Brain System: A Dynamic Adaptive Convolutional Quorum Voting Approach for Variable-Length EEG Data.
Complexity,Wiley-Hindawi, 2020:1-9, 2020,SCI二区,IF：2.121
(Abstract)
The brain is a complex and dynamic system, consisting of interacting sets and the temporal evolution of these sets. Electroencephalogram (EEG) recordings of brain activity play a vital role to decode the cognitive process of human beings in learning
research and application areas. In the real world, people react to stimuli differently, and the duration of brain activities varies
between individuals. ,erefore, the length of EEG recordings in trials gathered in the experiment is variable. However, current
approaches either fix the length of EEG recordings in each trial which would lose information hidden in the data or use the sliding
window which would consume large computation on overlapped parts of slices. In this paper, we propose TOO (Traverse Only
Once), a new approach for processing variable-length EEG trial data. TOO is a convolutional quorum voting approach that breaks
the fixed structure of the model through convolutional implementation of sliding windows and the replacement of the fully
connected layer by the 1 × 1 convolutional layer. Each output cell generated from 1 × 1 convolutional layer corresponds to each
slice created by a sliding time window, which reflects changes in cognitive states. ,en, TOO employs quorum voting on output
cells and determines the cognitive state representing the entire single trial. Our approach provides an adaptive model for trials of
different lengths with traversing EEG data of each trial only once to recognize cognitive states. We design and implement a
cognitive experiment and obtain EEG data. Using the data collecting from this experiment, we conducted an evaluation to
compare TOO with a state-of-art sliding window end-to-end approach. ,e results show that TOO yields a good accuracy
(83.58%) at the trial level with a much lower computation (11.16%). It also has the potential to be used in variable signal processing
in other application areas.
Zhou, Y., Xu, T., Li, S., Shi R.
Beyond Engagement: an EEG-based Methodology for Assessing User's Confusion in Educational Game.
Universal Access in the Information Society, Springer,18(3):551–563, 2019,SCI三区,IF：2.629
(Abstract)
Confusion is an emotion, which may occur when the learner is confronting inconsistence between new knowledge and existing cognitive structure, or reasoning for solving the puzzle and problem. Although confusion is not pleasant, it is necessary
for the learner to engage in understanding and deep learning. Consequently, confusion assessment has attracted increased
interest in e-learning. However, current studies have targeted no further than engagement detection and measurement, while
there is lack of studies in investigating cognitive and emotional aspects beyond engagement in the context of game-based
learning. To quantify confused states in logic reasoning in game-based learning, we propose an EEG-based methodology
for assessing the user’s confusion using the OpenBCI device with 8 channels. In the complicated context of game play, it is
difcult, and sometimes impossible, to collect the ground truth of the data in real tasks. To solve this issue, this work leverages cross-task and cross-subject methods to build a classifer, that is, training on the data of one standardized cognitive test
paradigm (Raven’s test) and testing on the data of real tasks in game play (Sokoban Game). It provides a new possibility
to create a classifer based on a small dataset. We also employ the end-to-end algorithm of deep learning in machine learning. Results showed the feasibility of this proposal in the task variation of the classifer, with an accuracy of 91.04%. The
proposed EEG-based methodology is suitable to analyze learners’ confusion on the long game-play duration and has a good
adaption in real tasks.
Zhou, Y., Qi L., Raake A., Xu T.,Piekarska M., Zhang X.
User Attitudes and Behaviors towards Personalized Control of Privacy Settings on Smartphones.
Concurrency and Computation Practice and Experience, Wiley,31(22):e4884, 2018,SCI三区,IF：1.831
(Abstract)
The fine-grained access control has been proved to be a reliable tool to ensure preserving of privacy of end users. In fog computing, one of the challenges is to understand users' attitudes and
behaviors toward personalized control. However, few of studies have given a clear view on users'
perception of the burden of interactivity when they set complex privacy settings. To this end, we
conducted a user study including a lab study with 26 participants and an evaluation with 223
participants. From the lab study, we found that participants were satisfied with improved privacy settings but did not adapt well to complex personalized interfaces. We proposed effective
methods to assist users to balance between the full control and the additional interaction burden,
including sorting, recommendations, and establishing profiles. After this lab study, we organized
a survey evaluation additionally to explore users' current usage of privacy features. Results from
the evaluation showed that the principle reason that users failed to use privacy features was
that they were not appropriately aware of these features. A key conclusion is that privacy settings should not only let users take over the control of smartphones but also inform them of the
knowledge on privacy practices.
Zhou, Y., Xu, T., David, B., Chalon, R.
Interaction on-the-go: a fine-grained exploration on wearable PROCAM interfaces and gestures in mobile situations.
Universal Access in the Information Society,Springer,15(2):1-15, 2015,SCI三区,IF：2.629
(Abstract)
Wearable projector and camera (PROCAM)
interfaces, which provide a natural, intuitive and spatial
experience, have been studied for many years. However,
existing hand input research into such systems revolved
around investigations into stable settings such as sitting or
standing, not fully satisfying interaction requirements in
sophisticated real life, especially when people are moving.
Besides, increasingly more mobile phone users use their
phones while walking. As a mobile computing device, the
wearable PROCAM system should allow for the fact that
mobility could influence usability and user experience.
This paper proposes a wearable PROCAM system, with
which the user can interact by inputting with finger gestures like the hover gesture and the pinch gesture on projected surfaces. A lab-based evaluation was organized,
which mainly compared two gestures (the pinch gesture
and the hover gesture) in three situations (sitting, standing
and walking) to find out: (1) How and to what degree does
mobility influence different gesture inputs? Are there any
significant differences between gesture inputs in different
settings? (2) What reasons cause these differences? (3)
What do people think about the configuration in such
systems and to what extent does the manual focus impact
such interactions? From qualitative and quantitative points
of view, the main findings imply that mobility impacts
gesture interactions in varying degrees. The pinch gesture
undergoes less influence than the hover gesture in mobile
settings. Both gestures were impacted more in walking
state than in sitting and standing states by all four negative
factors (lack of coordination, jittering hand effect, tired
forearms and extra attention paid). Manual focus influenced mobile projection interaction. Based on the findings,
implications are discussed for the design of a mobile projection interface with gestures.
Zhou, Y., Xu, T., David, B., Chalon, R.
Innovative Wearable Interfaces: An Exploratory Analysis of Paper-based Interfaces with Camera-glasses Device Unit.
Personal and Ubiquitous Computing, Springer,18(4):835-849, 2013,SCI二区,IF：3.006
(Abstract)
The new ubiquitous interaction methods change
people’s life and facilitate their tasks in everyday life and in
the workplace, enabling people to access their personal data as
well as public resources at any time and in any place. We
found two solutions to enable ubiquitous interaction and put a
stop to the limits imposed by the desktop mode: namely
nomadism and mobility. Based on these two solutions, we
have proposed three interfaces (Zhou et al. in HCI international 2011: human–computer interaction. Interaction techniques and environments, Springer, Berlin, pp 500–509,
2011): in-environment interface (IEI), environment dependent interface (EDI), and environment independent interface
(EII). In this paper, we first discuss an overview of IEI, EDI,
and EII, before excluding IEI and focusing on EDI and EII,
their background, and distinct characteristics. We also propose a continuum from physical paper-based interface to
digital projected interface in relation with EDI and EII. Then,
to validate EDI and EII concepts, we design and implement a
MobilePaperAccess system, which is a wearable cameraglasses system with paper-based interface and original input
techniques allowing mobile interaction. Furthermore, we
discuss the evaluation of the MobilePaperAccess system; we
compare two interfaces (EDI and EII) and three input techniques (finger input, mask input, and page input) to test the
feasibility and usability of this system. Both the quantitative
and qualitative results are reported and discussed. Finally, we
provide the prospects and our future work for improving the
current approaches.
Xu, T. Zhou, Y., Zhu, J
New Advances and Challenges of Fall Detection Systems: A Survey.
Applied Science,MDPI, 8(3):418, 2018,SCI二区,IF：2.838
(Abstract)
Falling, as one of the main harm threats to the elderly, has drawn researchers’ attentions
and has always been one of the most valuable research topics in the daily health-care for the elderly
in last two decades. Before 2014, several researchers reviewed the development of fall detection,
presented issues and challenges, and navigated the direction for the study in the future. With smart
sensors and Internet of Things (IoT) developing rapidly, this field has made great progress. However,
there is a lack of a review and discussion on novel sensors, technologies and algorithms introduced
and employed from 2014, as well as the emerging challenges and new issues. To bridge this gap,
we present an overview of fall detection research and discuss the core research questions on this
topic. A total of 6830 related documents were collected and analyzed based on the key words.
Among these documents, the twenty most influential and highly cited articles are selected and
discussed profoundly from three perspectives: sensors, algorithms and performance. The findings
would assist researchers in understanding current developments and barriers in the systems of
fall detection. Although researchers achieve fruitful work and progress, this research domain still
confronts challenges on theories and practice. In the near future, the new solutions based on advanced
IoT will sustainably urge the development to prevent falling injuries.
Xu, T. Zhou, Y.
Elders’ fall detection based on biomechanical features using depth camera.
International Journal of Wavelets, Multiresolution and Information Processing, World Scientific Publishing Company,16(2):1840005, 2018,SCI四区,IF：1.276
(Abstract)
An accidental fall poses a serious threat to the health of the elderly. With the advances
of technology, an increased number of surveillance systems have been installed in the
elderly home to help medical staffs find the elderly at risk. Based on the study of human
biomechanical equilibrium, we proposed a fall detection method based on 3D skeleton
data obtained from the Microsoft Kinect. This method leverages the accelerated velocity
of Center of Mass (COM) of different body segments and the skeleton data as main
biomechanical features, and adopts Long Short-Term Memory networks (LSTM) for fall
detection. Compared with other fall detection methods, it does not require older people
to wear any other sensors and can protect the privacy of the elderly. According to the
experiment to validate our method using the existing database, we found that it could
efficiently detect the fall behaviors. Our method provides a feasible solution for the fall
detection that can be applied at homes of the elderly.
Xu, T. Zhou, Y.
Fall prediction based on biomechanics equilibrium using Kinect.
International Journal of Distributed Sensor Networks,SAGE, 13(4) ,2017,SCI三区,IF：1.938
(Abstract)
The fall is one of the most important research fields of solitary elder healthcare at home based on Internet of Things
technology. Current studies mainly focus on the fall detection, which helps medical staffs bring a fallen elder out of danger in time. However, it neither predicts a fall nor provides an effective protection against a fall. This article studies the
fall prediction based on human biomechanics equilibrium and body posture characteristics through analyzing three-dimensional skeleton joints data from the depth camera sensor Kinect. The research includes building a human bionic
mass model using skeleton joints data from Kinect, determining human balance state, and proposing a fall prediction
algorithm based on recurrent neural networks by unbalanced posture features. We evaluate the model and algorithm on
an open database. The performance indicates that the fall prediction algorithm by studying human biomechanics can predict a fall (91.7%) and provide a certain amount of time (333 ms) before the elder injuring (hitting the floor). This work
provides a technical basis and a data analytics approach for the fall protection.
Xu, T. Zhou, Y., David, B., Chalon, R.
A smart brain: an intelligent context inference engine for context-aware middleware.
International Journal of Sensor Networks,Inderscience, 22(3):145-157, 2016,SCI四区,IF：1.264
(Abstract)
Recently, ‘activity’ context draws increased attention from researchers in context
awareness. Existing context-aware middleware usually employ the rule-based method to, which
is easy to build and also intuitive to work with. However, this method is fragile, not flexible
enough, and is inadequate to support diverse types of tasks. In this paper, we surveyed the related
literature in premier conferences over the past decade, reviewed the main activity context
recognition methods, and summarised their three main facets: basic activity inference, dynamic
activity analysis, and future activity recommendation. Based on our previous work, we then
proposed an intelligent inference engine for our context-aware middleware. Besides satisfying
requirements for checking context consistency, our inference engine integrates the three methods
for activity context recognition to provide a solution for all facets of activity context recognition
based on our context-aware middleware.
B Dong, J Bai, T Xu, Y Zhou.
Large language models in education: A systematic review.
2024 6th International Conference on Computer Science and Technologies in Education (CSTE)，IEEE,EI
(Abstract)
Large Language Models (LLMs) refer to a type of generative artificial intelligence model that produces responses to natural language input. The purpose of this study is to analyze the current application status of LLMs in the field of education through a systematic review of the literature. Data were sourced from three databases: Web of Science, ERIC, and Google Scholar. The study includes 94 documents, analyzed from both qualitative and quantitative perspectives. The results show that large language models have great potential in the field of education, specifically in generating medical content, serving as an English learning assistant, assisting academic research, and evaluating the quality of tests, etc. However, there are still potential dangers such as hindering the development of critical thinking, creating academic integrity crises, and ethical and moral challenges. These findings showed the current application status of LLMs in education, laying the groundwork to inspire future research.
Zhang,L., Guo,Y.,Ma,Q., Zhou,Y.
An Exploratory Study on Learning in 3D Multi-user Virtual Environments.
2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops(VRW),IEEE,EI
Chen,Q.,Zhang,L.,Dong,B.,Zhou,Y.
Interactive Cues on Geometry Learning in a Virtual Reality Environment for K-12 Education.
2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops(VRW),IEEE,EI
Deng, L., Zhou, Y., Cheng, T., Liu, X., Xu, T., Wang, X.
My English Teachers Are Not Human but I Like Them: Research on Virtual Teacher Self-study Learning System in K12.
International Conference on Human-Computer Interaction, Springer, Cham, 13329:176-187, 2022,EI
(Abstract)
English is the most used second language in the world. Mastering vocabulary is the prerequisite for learning English well. In middle
school, students direct their own studying outside the classroom to facilitate memorizing and understanding. However, the lack of teacher guidance in the self-study process will negatively affect learning. Producing
video lecture materials including talking teachers costs time, energy, and
money. The development of artificial intelligence makes it possible to create such materials automatically. However, it is still unclear whether a
system embedded with AI-generated virtual teachers can facilitate self-study compared with the system with merely the speech in self-study
in K12. To address this gap, we conducted a user study with 56 high
school students, collecting learning outcomes, user experience, and learning experience. Results showed that the virtual teacher helps to improve
the student’s English learning performance both in retention and transfer. Participants reported high user and learning experience. Our findings
shed light on the use of virtual teachers for self-study students in K12.
Bai, J., Zhang, H., Chen, Q., Cheng, X., Zhou, Y.
Technical Supports and Emotional Design in Digital Picture Books for Children: A Review.
The 13th International Conference on Ambient Systems, Networks and Technologies, Elsevier B.V.,201: 174-180, 2022,EI
(Abstract)
In recent years, the digital picture book has been an increasingly important reading and writing medium for children. Research
on the effects of digital picture books on learning has produced mixed results. As a potential tool, some researchers found that
such books could foster and scaffold for developing emergent literacy in the early childhood education. However, some still have
the skeptical attitude toward integrating the technology in picture books. This survey reviews techniques and emotional design that
have been applied in picture books. We also compared and discussed different types of picture books.
Zhang, L., Liu, X., Chen, Q., Zhou, Y., Xu, T.
EyeBox: A Toolbox based on Python3 for Eye Movement Analysis.
The 13th International Conference on Ambient Systems, Networks and Technologies, Elsevier B.V.,201: 166-173, 2022,EI
(Abstract)
Eye tracking technology can reflect human attention and cognition, widely used as a research tool. To analyze eye movement data,
users need to determine a specific area known as areas of interests (AOIs). Although existing tools offer dynamic AOIs functions to
process visual behavior on moving stimuli, they may ask users to use markers to specify contours of moving stimuli in the physical
environment or define AOIs manually on screen. This paper proposes a toolbox named Eyebox to 1) recognize dynamic AOIs
automatically based on SIFT and extract eye movement indicators, as well as 2) draw fixations. We also design a user-friendly
interface for this toolbox. Eyebox currently supports processing data recorded from the Pupil Core device. We compared results
processed by manual with by Eyebox in a custom eye-tracking dataset to evaluate this toolbox. The accuracy of 3/4 data for AOI1
is above 90%, and the accuracy of 4/5 data for AOI2 is higher than 90%. Finally, we conducted a user study to test the usability
and user experience of EyeBox.
Tao, Y., Zhang, G., Zhang, D., Wang, F., Zhou, Y., Xu, T.
Exploring Persona Characteristics in Learning: A Review Study of Pedagogical Agents.
The 13th International Conference on Ambient Systems, Networks and Technologies, Elsevier B.V.,201: 87-94, 2022,EI
(Abstract)
In digital interactive learning environments, pedagogical agents are digital characters that assist instruction, providing learners
with coaching, feedback, as well as emotional and social support. Recently, increased work has explored how animated, actual, or
artificial pedagogical agents impact learning. Some studies found that agents promote learning, while some revealed that agents
increased the workload and caused learners to lose focus. Especially poorly designed pedagogical agents would aggravate learners’
external cognitive load, negatively affecting learning performance and experience. This work reviewed how features of pedagogical agent roles affect learners’ retention and transfer performance, as well as the learning experience. It extracted five features,
including appearance, gender, facial expression, sound, and movements, then discussed their impact. The survey concludes with
the implications for designing effective agents.
Chen, Q., Deng, L., Xu, T., Zhou, Y.
Visualized Cues for Enhancing Spatial Ability Training in Virtual Reality.(PPT)
2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops , IEEE, 299-300, 2022,EI
(Abstract)
There is a growing research interest in training spatial ability in Virtual Reality (VR), which could provide a fully interactive learning
environment. However, existing spatial ability VR training systems
lack cues to assist mental rotation and visualization. Learners may
suffer from insufficient assistance to discover spatial reasoning rules
in the training process. The aims of most visual cues for pedagogical
purposes are to draw the viewer’s attention, which can not meet
the requirements of spatial training tasks. To address this issue, we
propose three cues for learners to succeed in spatial tasks: color,
rotation path, and circle. These visual cues provide supplementary
spatial information and just-in-time explanations. They help distinguish the different parts of the solid, visualize the spatial information
like rotation and direction, and emphasize the key points to solve
the problems.
Liu, X., Zhang, S., Xu, T., Zhou, Y.
Improving Language Learning by an Interact-to-Learn Desktop VR Application: A Case Study with Peinture. (PPT)
2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, IEEE, 267-270, 2022,EI
(Abstract)
Mastering vocabulary is a daunting task for second language learners. Desktop Virtual Reality (VR) can inspire and engage learners
with interact-to-learn activities to facilitate memorizing vocabulary.
This paper proposes a desktop VR application for color vocabulary
learning called Peinture. Peinture consists of a lecturing video and
an interactive coloring virtual space. The talking teacher in the lecturing video is generated automatically through text-to-speech and
lip synthesis approaches. Learners can paint and interact in a theme
when or after lecture learning. A preliminary user study based on
performance, self-report and eye tracking data analysis shows that
interact-to-learn activities provided by Peinture were beneficial for
second language learning. However, there is still considerable room
to upgrade such educational VR.
Zhang, G., Li, S., Wang, J., Zhou, Y., Xu, T.
A Semi-automatic Feature Fusion Model for EEG-based Emotion Recognition.
2021 27th International Conference on Mechatronics and Machine Vision in Practice, IEEE,726-731, 2021,EI
(Abstract)
Electroencephalogram (EEG) is usually used to
study cognitive activities, which have different temporal,
frequency-domain features. Scientists attempted to find crucial
features to improve recognition accuracy but challenging. This
paper proposed a novel confused emotion recognition method
based on EEG, which combine automatic feature extraction
(deep learning) and knowledge-based feature extraction. To
evaluate our method, we designed an experiment to collect
data, the basic idea of which is to induce the confused emotion
based on the English listening test. The results show that
our method performs better in experiments than Convolution
Neural Networks(CNN) and Support Vector Machine (SVM).
Xu, T., Wang, X., Wang, J., Zhou, Y.
From Textbook to Teacher: an Adaptive Intelligent Tutoring System Based on BCI.
2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society,IEEE, 7621-7624, 2021,EI
(Abstract)
In this work, we propose FT3
, an adaptive intelligent tutoring system based on Brain Computer Interface(BCI).
It can automatically generate different difficulty levels of
lecturing video with teachers from textbook adapting to student
engagement measured by BCI. Most current studies employ
animated images to create pedagogical agents in such adaptive
learning environments. However, evidence suggests that human
teacher video brings a better learning experience than animated
images. We design a virtual teacher generation engine consisting
of text-to-speech (TTS) and lip synthesis method, being able
to generate high-quality adaptive lecturing clips of talking
teachers with accurate lip sync merely based on a textbook
and teacher’s photo. We propose a BCI to measure engagement,
serving as an indicator for adaptively generating appropriate
lecturing videos. We conduct a preliminary study to build and
evaluate FT3
. Results verify that FT3 can generate synced
lecturing videos, and provide proper levels of learning content
with an accuracy of 73.33%.
Xu, T., Zhou, Y., Wang Y., Zhao Z., Li S.
Guess or Not? A Brain-Computer Interface Using EEG Signals for Revealing the Secret behind Scores.(VEDIO)
CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, ACM,LBW1210:1-6,2019,EI
(Abstract)
Now examinations and scores serve as the main criterion for a student’s academic performance.
However, students use guessing strategies to improve the chances of choosing the right answer in a
test. Therefore, scores do not reflect actual levels of the student’s knowledge and skills. In this paper,
we propose a brain-computer interface (BCI) to estimate whether a student guesses on a test question
or masters it when s/he chooses the right answer in logic reasoning. To build this BCI, we first define the “Guessing” and employ Raven’s Progressive Matrices as logic tests in the experiment to collect
EEG signals, then we propose a sliding time-window with quorum-based voting (STQV) approach to
recognize the state of “Guessing” or “Understanding”, together with FBCSP and end-to-end ConvNet
classification algorithms. Results show that this BCI yields an accuracy of 83.71% and achieves a good
performance in distinguishing “Guessing” from “Understanding”.
Zhou, Y., Xu, T., Li S., Li S.
Confusion State Induction and EEG-based Detection in Learning English.
In 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE,3290-3293, 2018,EI
(Abstract)
Confusion, as an affective state, has been proved
beneficial for learning, although this emotion is always mentioned as negative affect. Confusion causes the learner to
solve the problem and overcome difficulties in order to restore
the cognitive equilibrium. Once the confusion is successfully
resolved, a deeper learning is generated. Therefore, quantifying
and visualizing the confusion that occurs in learning as well
as intervening has gained great interest by researchers. Among
these researches, triggering confusion precisely and detecting it
is the critical step and underlies other studies. In this paper, we
explored the induction of confusion states and the feasibility
of detecting confusion using EEG as a first step towards
an EEG-based Brain Computer Interface for monitoring the
confusion and intervening in the learning. 16 participants
EEG data were recorded and used. Our experiment design
to induce confusion was based on tests of Raven’s Standard
Progressive Matrices. Each confusing and not-confusing test
item was presented during 15 seconds and the raw EEG data
was collected via Emotiv headset. To detect the confusion
emotion in learning, we propose an end-to-end EEG analysis
method. End-to-end classification of Deep Learning in Machine
Learning has revolutionized computer vision, which has gained
interest to adopt this method to EEG analysis. The result of
this preliminary study was promising, which showed a 71.36%
accuracy in classifying users’ confused and unconfused states
when they are inferring the rules in the tests.
Xu, T., Zhou, Y., Wang Z., Peng, Y.
Learning Emotions EEG-based Recognition and Brain Activity: A Survey Study on BCI for Intelligent Tutoring System.
The 9th International Conference on Ambient Systems, Networks and Technologies ,Elsevier B.V., 130:376-382, 2018,EI
(Abstract)
Learners experience emotions in a variety of valence and arousal in learning, which impacts the cognitive process and the success of
learning. Learning emotions research has a wide range of benefits from improving learning outcomes and experience in Intelligent
Tutoring System (ITS), as well as increasing operation and work productivity. This survey reviews techniques that have been used
to measure emotions and theories for modeling emotions. It investigates EEG-based Brain-Computer Interaction (BCI) of general
and learning emotion recognition. The induction methods of learning emotions and related issues are also included and discussed.
The survey concludes with challenges for further learning emotion research.
Zhou, Y., Xu, T., Zhi X., Wang Z.
Learning in Doing: A Model of Design and Assessment for Using New Interaction in Educational Game.
International Conference on Learning and Collaboration Technologies LCT 2018: Learning and Collaboration Technologies, Learning and Teaching, Springer, Cham,10925:225-236, 2018,EI
(Abstract)
To put into practice what has been learned is considered as one of the most important education objectives. In the traditional class, it is difficult for learners to engage in practices since teachers usually convey knowledge and experience by speaking or simple demonstrations, like using slides and videos. This brings obstacles for learners to apply what they have learned to solve real problems. Educational game with the novel input and out technologies is one of the solutions for learners to engage in the learning activities. It can not only effectively encourage learners to learn positively and vividly, promote learning interests and motivation, and enhance the engagement, but also improve their imagination, learning performance and other learning behaviors. This paper first discusses the learning theories related to educational game. Then a literature review is conducted by collecting data on the topics of assessment of educational game with new interaction. Next, we propose a model to guide the design and the evaluation of educational game due to the missing studies. This model underlies an educational game to foster the garbage classification learning with virtual 3D output and natural gesture input using Leap Motion, which is a tracking device of hands and objects. To analyze users’ learning behaviors and evaluate their performance and experience, we conduct an evaluation with 22 college learners. Results showed that the use of the natural interaction not only made the learning interesting and fostered the engagement, but also improved the absorption of knowledge in practices. Finally, a discussion of the challenges and future directions is presented.
Zhou, Y., Ji S., Xu, T., Wang Z.
Promoting Knowledge Construction: A Model for Using Virtual Reality Interaction to Enhance Learning.
The 9th International Conference on Ambient Systems, Networks and Technologies,Elsevier B.V., 130:239-246, 2018,EI
(Abstract)
VR technologies, offering powerful immersion and rich interaction, have gained great interest from researchers and practitioners in the field of education. However, current learning theories and models either mainly take into account the technology perspectives,
or focus more on the pedagogy. In this paper, we propose a learning model benefiting from both the Human-Computer Interaction
aspects and pedagogical aspects. This model takes full account of the impact of different factors including pedagogical contexts,
VR roles and scenarios, and output specifications, which would be combined to inform the design and realize VR education
applications. Based on this model, we design and implement an educational application of computer assembly under virtual
reality using HTC Vive, which is a headset providing immersion experience. To analyze users’ learning behaviors and evaluate
their performance and experience, we conduct an evaluation with 32 college students as participants. We design a questionnaire
including usability tests and emotion state measures. Results showed that our proposed learning mode gave a good guidance for
informing the design and use of VR-supported learning application. The use of the natural interaction not only makes the learning
interesting and fosters the engagement, but also improves the construction of knowledge in practices.
Zhou, Y., Xu, T., Cai, Y., Wu, X., Dong B.
Monitoring cognitive workload in online videos learning through an EEG-based brain-computer interface.
International Conference on Learning and Collaboration Technologies LCT 2017: Learning and Collaboration Technologies, Learning and Teaching,Springer, Cham, 10295:64-73, 2017,EI
(Abstract)
Student cognitive state is one of the crucial factors determing
successful learning [1]. The research community related to education and
computer science has developed various approches for describing and monitoring
learning cognitive states. Assessing cognitive states in digital environment makes
it possible to supply adaptive instruction and personalized learning for student.
This assessment has the same function as the instructor in a real-world classroom
observing and adjusting the speed and contents of the lecture in line with students’
cognitive states. The goal is to refocus students’ interest and engagement, making
the instruction efficiently. In recent years, increased researches have focused on
various measures of cognitive states, among which physiological measures are
able to monitor in a real-time, especially electroencephalography (EEG) based
brain activity measures. The cognitive workload that students experience while
learning instructional materials determines success in learning. In this work, we
design and propose a real-time passive Brain-Computer Interaction (BCI) system
to monitor the cognitive workload using EEG-based headset Emotiv Epoc+,
which is feasible for working in the online digital environment like Massive Open
Online Courses (MOOCs). We choose two electrodes to pick up original EEG
signals, which are highly relevant to the workload. The current prototype is able
to record EEG signals and classify levels of cognitive load when students
watching online course videos. This prototype is based on two layers, using
machine learning approaches for classification.
Xu, T. Zhou, Y., David, B., Chalon, R.
Supporting Activity Context Recognition in Context-aware Middleware.
Workshops at the Twenty-Seventh AAAI Conference on Artificial Intelligence (AAAI’13), Bellevue, Washington, USA, 61-70,(2013),AI Access Foundation,EI
(Abstract)
Context-aware middleware is considered as an efficient solution to develop context-aware application, which provides a feasible development platform integrating various sensors and new technologies. With the development of sensors, the research work on the context shift from “location” to “activity” gradually. Then it puts forward a new requirement for context-aware middleware: activity context recognition. In this paper, we first survey the related literature in premier conferences over the past decade, review the main methods of activity context recognition, and conclude its three main facets: basic activity inference, dynamic activity analysis and future activity recommendation. We then propose an intelligent inference engine integrating three popular methods solving three facets mentioned above respectively, which is based on our context-aware middleware. Finally, to evaluate the feasibility of our proposal, we implement two scenarios: bus stop scenario and domestic activity application.
Xu, T., David, B., Chalon, R., Zhou, Y.
A Context-aware Middleware for Ambient Intelligence.
ACM/IFIP/USENIX 12th International Middleware Conference (Middleware 2011), Poster session, Lisbon, Portugal,10:1-2, 2011,ACM,EI
(Abstract)
By the acronym MOCOCO we refer to our view of ambient
intelligence (AmI), pointing out three main characteristics:
Mobility, Contextualization and Collaboration. The ambient
intelligence is a challenging research area focusing on ubiquitous
computing, profiling practices, context awareness, and human-centric computing and interaction design. We are concretizing our
approach in a platform called IMERA (French acronym for
Computer Augmented Environment for Mobile Interaction). In
order to make work together several sensors, actuators and mobile
smart devices, the need of a context-aware middleware for this
platform is obvious. In this paper, we present main objectives and
solution principles concretized in a context-aware middleware
based on hybrid reasoning engine (the ontology reasoning and the
decision tree reasoning), which retrieves efficiently high-level
contexts from raw data. This platform provides an environment
for rapid prototyping of context aware services in Ambient
Intelligent (AmI).
加入我们
主要研究方向
本团队聚焦人工智能、虚拟现实与脑机接口等前沿技术在教育领域的创新应用，融合计算机科学、教育技术学、认知心理学与脑科学，探索智能技术如何改变知识呈现、学习交互与认知过程。研究兼顾人工智能算法与交互系统开发、学习科学理论探索以及真实教育场景中的实证研究，欢迎对人工智能、教育技术、智能交互和学习科学感兴趣的本科生、硕士生及博士生加入。
1. 生成式人工智能驱动的教学资源设计与学习机制
研究关键词：生成式人工智能（AIGC）、AI教学视频、数字教师、多模态学习、AI Agent、学习科学
生成式人工智能正在改变传统教学资源的设计与呈现方式。从能够自主讲解知识的AI数字教师，到具有丰富表现形式的智能教学视频，再到能够根据学习者需求提供个性化指导的AI学习智能体，人工智能正在为教学创新提供新的可能。
本方向重点研究如何利用生成式人工智能设计更有效、更具吸引力的教学资源，以及这些资源如何影响学习者的认知、情感和学习行为。通过结合学习科学理论、人工智能技术与实验研究方法，探索AI生成内容的拟人化特征、多模态呈现方式和社会交互线索对学习成效、认知负荷、社会临场感、信任及学习体验的影响，揭示其作用机制与适用边界。
同时，探索大语言模型与智能体技术在教学资源自动生成、个性化学习支持、自适应反馈及智能教学系统中的创新应用，推动从“AI生成教学内容”向“AI理解并支持学习过程”的转变。
2. 智能交互与虚拟现实在教育中的应用
研究关键词：虚拟现实（VR/XR）、人机交互（HCI）、智能学习环境、眼动追踪、空间认知、协作学习
虚拟现实与智能交互技术能够突破传统课堂在空间、时间和交互方式上的限制，使学习者进入可探索、可操作、可协作的沉浸式学习环境，为抽象知识理解、复杂技能训练和高阶认知能力培养提供新的途径。
本方向重点研究如何设计符合人类认知规律的智能交互与沉浸式学习环境，以及不同交互方式如何影响学习者的认知加工、学习行为与能力发展。围绕沉浸式虚拟现实、多人协作学习、空间认知与空间能力训练等场景，研究虚拟环境设计、交互方式、智能反馈及学习者个体差异对学习成效、认知负荷、协作过程与知识迁移的影响。
进一步融合大语言模型、眼动追踪、行为分析与自适应交互技术，开发能够感知学习者状态、动态调整学习任务与交互方式的智能虚拟学习环境，探索从传统沉浸式学习向智能化、个性化虚拟学习的演进。
3. 教育脑机接口与多模态学习状态感知
研究关键词：脑机接口（BCI）、脑电信号（EEG）、眼动追踪、多模态人工智能、认知状态识别、情感计算
学习不仅表现为外在行为的变化，还伴随着复杂的注意、记忆、认知负荷与情绪变化。如何理解学习过程中难以直接观察的认知与情感状态，是智能教育与学习科学的重要研究问题。
本方向融合信息科学、教育学、心理学与脑科学，从“行为—心理—生理—神经”多层次研究学习者在复杂学习任务和深度认知加工过程中的状态变化与内在机制。通过脑电信号、眼动数据、生理信号及学习行为等多模态信息，探索注意力、认知负荷、情绪、学习投入等状态的动态识别与建模方法，研究不同模态之间的关联规律及其对学习过程的解释价值。
在此基础上，探索基于机器学习、深度学习与多模态信息融合的学习状态识别技术，以及脑机接口和神经反馈在智能教育中的应用，构建能够感知学习者认知与情感变化、提供个性化支持的智能学习系统，为自适应教学、人机协同学习与学习过程评价提供新的方法和技术路径。
欢迎加入我们的研究团队
我们欢迎具有教育技术学、计算机科学、人工智能、心理学及相关学科背景，对前沿技术与教育创新充满兴趣的学生加入。
无论你希望开发具有智能交互能力的AI教学系统、探索沉浸式虚拟现实学习环境，还是利用脑电、眼动和人工智能揭示人类学习的奥秘，都可以在这里找到值得深入探索的研究问题。
团队鼓励跨学科合作，重视理论探索、技术创新与实验研究相结合，支持学生根据自身兴趣逐步形成具有个人特色的研究方向。
数据库与工具
Confusion EEG DB