Research on a DeepSeek-Based Multimodal Human-Computer Interaction System
Main Article Content
Keywords
DeepSeek, multimodal large model, in-vehicle human computer interaction, contextual reasoning, implicit intention recognition, RAG retrieval augmentation
Abstract
The existing general multimodal large models based on DeepSeek have severe limitations when directly applied to in-vehicle environments, such as fragmented static input, long-context forgetting, absence of vertical-domain knowledge, and failure to comprehend the implied intentions of the drivers. This paper will present a multimodal contextual reasoning human-computer interaction paradigm for intelligent vehicle cockpits. Four main directions exist in order to adapt DeepSeek-VL to specific scenarios, namely, temporal context modeling, dual-layer long-term memory expansion, RAG retrieval augmentation with an automotive vertical knowledge base, and joint environmental-driver causative contextual reasoning. They are tested on four publicly available datasets UTA-RLDD, DMD, InCarEmo, and CMU-MOSEI datasets as well as a self-built dataset, with native DeepSeek-VL, LLaVA-1.5, and Qwen-VL serving as the baseline. The findings indicate that the proposed scheme attains an implicit intention recognition rate of 86.4%, which is 34.1 percentage points higher than that of the native model; a driving risk recall of 91.1; and a decrease in the hallucination rate of 23.5 percent to 4.6 percent. The ablation experiments prove the independent contribution of each module quantitatively and the subjective evaluation of users shows major improvements in all aspects.
References
- [1] China Association of Automobile Manufacturers. (2023). White paper on the development of China’s intelligent cockpit industry in 2023.
- [2] Keqiang Li, Yifan Dai, Shuwei Zhang. (2022). Current status and development trends of human-computer interaction technology for intelligent connected vehicles. Automotive Engineering, 44 (10), 1467-1478.
- [3] Hui Zhang, Wujun Wang. (2022). Development status and trends of human-computer interaction technology in intelligent cockpits. Journal of Automotive Engineering, 12 (3), 317-326.
- [4] Hua Li, Xiaofeng Zhao. (2024). A survey of multimodal large model technology. Computer Engineering and Applications, 60 (5), 1-18.
- [5] Zhou, K., Liu, Z., Duan, N., et al. (2023). Towards unified multi-modal learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (9), 10847-10868.
- [6] Jian Wang, Yangli. (2021). Research progress on driver fatigue state detection technology. China Safety Science Journal, 31 (8), 1-9.
- [7] Xiangmu Zhao, Junyan Ma, Kai Zhang. (2022). Research progress and prospects of driver fatigue state detection technology. China Journal of Highway and Transport, 35 (8), 1-20.
- [8] Chang Liu, Lei Zhang. (2024). A survey of retrieval-augmented generation (RAG) technology. Computer Science, 51 (4), 1-15.
- [9] Jian Chen, Bo Li. (2024). A survey of retrieval-augmented generation technology. Journal of Computer Research and Development, 61 (2), 356-375.
- [10] DeepSeek-AI. (2024). DeepSeek-VL: Towards real-world vision-language understanding (arXiv:2403.05525).
- [11] Hao Zheng, Jie Sun. (2022). Research progress on multimodal affective computing in in-vehicle scenarios. Journal of Computer-Aided Design & Computer Graphics, 34 (10), 1523-1534.
- [12] Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (pp. 9459-9474).
- [13] Zhang, Y., Liu, Q., Wang, Z., et al. (2023). In-car voice interaction intent recognition based on BERT and context information. In Proceedings of the 2023 ACM International Conference on Multimodal Interaction (ICMI) (pp. 456-464). ACM.
- [14] Ming Zhou, Fang Liu. (2024). A survey of long-context processing technology for large models. Journal of Chinese Information Processing, 38 (3), 1-14.
- [15] Yang Yang, Hong Liu. (2023). Research progress on in-vehicle multimodal affective computing. Acta Automatica Sinica, 49 (6), 1123-1140.
- [16] Yu Chen, Tao Wu. (2023). Research on user experience of Chinese in-vehicle voice interaction. Chinese Journal of Ergonomics, 29 (2), 45-51.
- [17] Ghoddoosian, R., Galib, M., & Athitsos, V. (2019). A realistic dataset and baseline temporal model for early drowsiness detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 1-8).
- [18] Driver Monitoring Dataset (DMD). (2022). Joint Laboratory for Autonomous Driving Industry.
- [19] Harbin Institute of Technology Human-Computer Interaction Lab. (2023). InCarEmo Chinese in-vehicle emotion dataset.
- [20] Qun Liu, Yu Zhang. (2023). Research on intent recognition methods for in-vehicle voice interaction. Application Research of Computers, 40 (5), 1356-1361.
- [21] Zadeh, A., Liang, P. P., Mazumder, N., et al. (2018). Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (pp. 1-11).
- [22] National Technical Committee of Auto Standardization. (2020). GB/T 39268-2020 GB/T 39268-2020 Determination of fatigue driving for motor vehicle drivers. China Standards Press.
- [23] Houlsby, N., Giurgiu, A., Jastrzebski, S., et al. (2019). Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (pp. 2790-2799).
- [24] Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (pp. 5998-6008).
- [25] Wei Zhang, Chao Wang. (2024). A survey of lightweight fine-tuning techniques for multimodal large models. Computer Engineering, 50 (3), 1-12.
- [26] Chen, M., Liu, S., Li, Y., et al. (2024). Lightweight fine-tuning for multimodal large language models in autonomous driving scenarios. In 2024 IEEE Intelligent Vehicles Symposium (IV) (pp. 1234-1240). IEEE.
- [27] Hu, E. J., Shen, Y., Wallis, P., et al. (2022). LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (pp. 1-14).
- [28] Zhiyong Wu, Yicai. (2022). User experience evaluation system for Chinese in-vehicle voice interaction. Chinese Journal of Ergonomics, 28 (4), 32-38.
- [29] Wang, H., Zhang, R., & Chen, L. (2023). Context-aware active human-computer interaction for smart cockpits. In 2023 International Conference on Human-Computer Interaction (HCI) (pp. 345-356). Springer.
- [30] Appendix
- [31] Appendix A: Core Experimental Code Snippets and Related Experimental Figures
Article Sidebar
This work is licensed under a Creative Commons Attribution 4.0 International License.