Understanding and reasoning in complex scenes are regarded as key indicators for evaluating the intelligence level of robots. However, existing methods mainly focus on question answering tasks in static environments and are insufficient for addressing multi-step reasoning requirements in dynamic scenes. A robot scene understanding and reasoning method based on distributed knowledge encoding was proposed. A scene knowledge graph with a dual-layer structure of concepts and attributes was constructed to enhance scene understanding capability. Entity concepts were represented as attribute sets through distributed embedding techniques, which enables compact and computable semantic representation. In addition, a classifier was designed to infer the sequence of robot actions required to achieve scene transformation based on the initial scene and the target scene. The proposed distributed knowledge encoding method exhibits good interpretability. Multi-step scene transformation processes were decomposed into executable single-step action sequences, such that the reasoning complexity increases linearly with the number of transformation steps. Experiments were conducted on the public Trance dataset. The experimental results demonstrate that, compared with GPT-4o, the proposed method achieves superior performance in scene transformation sensitivity and multi-step reasoning capability. Furthermore, feasibility and robustness in real-world scenarios are validated on a real robotic platform.
WANG Hai-tao, ZHANG Shao-lin, JIANG Tian-yu, GE Yue-guang, CUI Shao-wei, WANG Shuo.
Distributed Knowledge Encoding for Scene Understanding and Reasoning in Robotics[J].
Science Technology and Engineering, 2026, 26(9): 3634-3641 DOI:10.12404/j.issn.1671-1815.2506683
HuangS, PonomarenkoI, JiangZ, et al. ManipVQA: injecting robotic affordance and physically grounded information into multi-modal large language models[C]// IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Abu Dhabi: IEEE, 2024: 7580-7587.
LiJincheng, DaiXuejing, YanRuiao. Gait recognition algorithm based on 3D-CNN and integrated Transformer[J]. Science Technology and Engineering, 2025, 25(17): 7276-7284.
HouYonghong, ZhengHaochun, GaoJiajun, et al. Zero-shot action recognition based on the CLIP model and a knowledge database[J]. Journal of Tianjin University (Science and Technology), 2025, 58(1): 91-100.
[9]
WilliamsC K I. Structured generative models for scene understanding[J]. International Journal of Computer Vision, 2025, 133(5): 2845-2867.
[10]
DriessD, XiaF, SajjadiM S M, et al. PaLM-E: an embodied multimodal language model[C]// International Conference on Machine Learning. Hawaii: PMLR, 2023: 8469-8488.
[11]
LiJ, LiD, SavareseS, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[C]// International Conference On Machine Learning. Hawaii: PMLR, 2023: 19730-19742.
[12]
GuQ, KuwajerwalaA, MorinS, et al. Conceptgraphs: open-vocabulary 3D scene graphs for perception and planning[C]// IEEE International Conference on Robotics and Automation (ICRA). Yokohama: IEEE, 2024: 5021-5028.
ChuXulong, ShiDongmei, LiuJinchang. The hot spot of intelligent robot research in China in recent 20 years-bibliometric analysis based on CiteSpace[J]. Science Technology and Engineering, 2023, 23(6): 2477-2484.
WangShuai, HeWenchun, WangFudi, et al. Research on question answering system based on large language model integrating knowledge graph and vector retrieval[J]. Science Technology and Engineering, 2024, 24(32): 13902-13910.
XiaYi, LanMingjing, ChenXiaohui, et al. Survey on explainable knowledge graph reasoning methods[J]. Chinese Journal of Network & Information Security, 2022, 8(5): 1-25.
[19]
RavichandranZ, PengL, HughesN, et al. Hierarchical representations and explicit memory: learning effective navigation policies on 3D scene graphs using graph neural networks[C]// International Conference on Robotics and Automation (ICRA). Philadelphia: IEEE, 2022: 9272-9279.
[20]
DarunaA, DasD, ChernovaS. Explainable knowledge graph embedding: inference reconciliation for knowledge inferences supporting robot actions[C]// IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Kyoto: IEEE, 2022: 1008-1015.
[21]
HongX, LanY, PangL, et al. Transformation driven visual reasoning[C]// Proceedings of the IEEE/CVF Conference on Computer Vision And Pattern Recognition. Nashville: IEEE, 2021: 6903-6912.
ZhangYu, FengYahan, DingQianhui. Multi-sentiment classification of music lyrics by incorporating Word2Vec word embedding multi-core convolutional neural networks[J]. Science Technology and Engineering, 2024, 24(20): 8598-8605.
[24]
LiangY, MaoW, WuC. Unsupervised incremental transfer learning with knowledge distillation for online remaining useful life prediction of rotating machinery[J]. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 2025, 239(1): 15-30.
[25]
CiniA, MariscaI, BianchiF M, et al. Scalable spatiotemporal graph neural networks[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Washington DC: AAAI, 2023, 37(6): 7218-7226.
[26]
HuangY, ZhouJ, KeP, et al. A skin-integrated multimodal haptic interface for immersive tactile feedback[J]. Nature Electronics, 2023, 6(12): 1020-1031.
[27]
YamasakiY, TokunagaM, SakaiY, et al. Effects of a force feedback function in a surgical robot on the suturing procedure[J]. Surgical Endoscopy, 2024, 38(3): 1222-1229.
PDF
(9831KB)
