融入先驗知識以增強動作識別中跨領域少樣本學習能力

Abstract

行動識別是視頻理解中的關鍵領域，通常需要大量的訓練數據。為了解決 這個問題，我們採用了少樣本學習方法。然而，這些方法主要設計用於同領域的 場景，當應用到現實世界的跨領域情況時，會面臨挑戰。在本研究中，我們引入 了一種新的數據表示方式——“軌跡 (Trajectory )”，以及一個“交叉相似性注意力 (CSA)塊”，它們都是基於行動識別的先驗知識，並且可以輕鬆地整合到現有的 少樣本學習方法中。 “軌跡 (Trajectory )”方法是一種新的骨骼數據表示方式，利用空間信息來彌補 由於視頻採樣導致的時間數據損失。這種方法使我們能夠使用更少的幀數和更少 的計算資源達到與更多幀數相比的結果。 CSA塊利用骨骼數據的獨特特性來增強 空間和時間相似性的比較，從而使度量學習能夠生成更好的嵌入。 我們還將視覺提示學習整合到新領域適應的微調過程中。我們的方法不僅在 開放數據集上表現出強大的性能，還在現實世界的場景中，如我們實驗室收集 的AIMS項目中的嬰兒行為數據，也展現了出色的表現。這突顯了它在解決具有有 限標籤數據的實際行動識別挑戰的實用應用性和潛力。

Action recognition, a critical domain in video understanding, typically requires a substantial amount of training data. To address this, we employ few-shot learning methods. However, these methods, primarily designed for same-domain scenarios, face challenges when applied to real-world, cross-domain situations. In this study, we introduce a novel data representation, ’Trajectory’, and a ’Cross-Similarity Attention (CSA) Block’, both informed by prior knowledge specific to action recognition and easily integrated into existing few-shot learning methods. The ’Trajectory’ method, a new data representation for skeleton data, leverages spatial information to compensate for the temporal data loss due to video sampling. This approach allows us to achieve comparable results to those obtained with more frames but with fewer frames and less computational resources. The CSA Block utilizes the unique characteristics of skeleton data for enhanced comparison of spatial and temporal similarities, enabling metric learning to generate better embedding. We also incorporate visual prompt learning for fine-tuning during the adaptation to new domains. Our method demonstrates robust performance not only on open datasets but also in real-world scenarios, such as the infant action data collected in our lab’s AIMS project. This underlines its practical applicability and potential in addressing real-world action recognition challenges with limited labeled data.