以人與物互動與圖型表徵自動分析營造場域之工人任務

Abstract

在建築專案管控中，有效追溯工人的任務範圍及落實程度需要能夠捕捉工人與物件之間複雜且動態的交互作用。然而，現有的人–物互動（Human–Object Interaction, HOI）方法仍受限於僅專注於單一動作、有限的物件尺度，以及不足的空間推理，使其難以應用於大型且多情境的建築環境。為解決這些問題，本研究提出一套工人任務可溯性偵測（Worker Accountability Monitoring, WAM）框架，結合卷積神經網路（CNN）與圖注意力網路（Graph Attention Networks, GATs），能夠偵測多種物件類別（混凝土、模板及鋼筋），並涵蓋三種尺度（大、中、小）及兩種空間情境（局部與全域）。該框架同時辨識互動行為（如綁紮、搬運、澆置）與非互動行為，提供工地活動的整體視角。實驗結果顯示，本方法在物件偵測中達到 mAP = 0.830，在 HOI 任務的局部與全域情境下分別取得 0.553 與 0.502 的 mAP，並顯著優於現有 HOI 基準方法。

本研究透過以下方式解決了現有 HOI 方法在構建中的關鍵局限性：（1）引入各種規模的對象表示（大、中、小）以更好地反映現實世界的情況，（2）整合局部和全局背景以改進空間推理，以及（3）實現每幀的多個同時交互以處理擁擠、複雜的環境。然而，現階段仍存在若干限制，包括資料集不平衡、施工任務涵蓋範圍有限、依賴靜態 2D 表徵，以及缺乏時序與幾何推理。為克服這些挑戰，本研究正持續拓展至「空間–姿態–時間」WAM 架構，結合圖神經網路與 Transformer 模組，以捕捉動作序列並強化精細互動推理。未來研究將進一步推動 WAM 朝向即時生產力監測、透過視覺–語言模型的自動化報告生成、結合 LBMS 的動態排程，以及人機協作施工流程的應用發展。綜合而言，本研究為基於電腦視覺的責任監測建立了嚴謹基礎，並為提升 WAM 的可擴展性、穩健性與現場實用性勾勒出清晰的研究藍圖。

Effectively monitoring worker accountability in construction project controls requires capturing complex and dynamic interactions between workers and objects. However, existing Human-Object Interaction (HOI) approaches remain constrained by their focus on single actions, limited object scales, and insufficient spatial reasoning, making them unsuitable for large-scale and multi-context construction environments. To address these gaps, this study introduces a Worker Accountability Monitoring (WAM) framework that integrates Convolutional Neural Networks with Graph Attention Networks to detect interactions across multiple object categories: concrete, formwork, and steel rebar at three scales (big, medium, small) and two spatial contexts (local and global). The framework recognizes both interactive actions (e.g., tying, transporting, pouring) and non-interactive behaviors, offering a holistic view of on-site activities. Experimental results demonstrate strong performance, achieving an mAP of 0.830 for object detection and HOI scores of 0.553 (local) and 0.502 (global), significantly outperforming representative HOI baselines.

This study addresses key limitations of existing HOI approaches in construction by: (1) introducing various scale object representations (big, medium, and small) to better reflect real-world conditions, (2) integrating local and global contexts for improved spatial reasoning, and (3) enabling multiple simultaneous interactions per frame to handle crowded, complex environments. However, limitations remain, including imbalanced datasets, restricted coverage of construction tasks, reliance on static 2D formulations, and limited temporal or geometric reasoning. To overcome these constraints, ongoing work extends WAM into a spatial-pose-temporal framework that combines graph neural networks and transformer-based modules for sequential action modeling and fine-grained interaction reasoning. Future research will further advance WAM toward real-time productivity monitoring, automated construction reporting via vision-language models, dynamic scheduling with LBMS, and human-robot collaboration in mixed crews. Collectively, this study establishes a rigorous foundation for vision-based accountability monitoring and outlines a clear trajectory for advancing scalable, robust, and field-ready WAM systems.