基於相機視角與注意力機制的多尺寸感受域特徵提取器

Abstract

隨著電腦視覺領域的發展，如何有效地描述影像資訊成為重要的課題。然而特徵提取就是其中一種被用來描述影像的方式，同時也被運用在許多任務，像是物件偵測、物件追蹤、圖像拼接、三維視覺重建及定位等任務。在過去有許多不同的特徵提取方式被提出來做應用，像是基於圖像紋理、邊緣、形狀等基本特徵並且利用梯度或差值等數學關係描述的手工特徵提取方式。又或者是近期較熱門基於深度學習的特徵描述方式，其利用多層卷積神經網路或是注意力機制，有效的去獲得局部及全局的影像資訊，並且利用一定量的資料協助訓練，讓神經網路最佳化其結果。以上兩種方式皆是常見的特徵描述方式，然而近期基於深度學習的方式在許多資料集上都有不錯的表現，所以我們會基於現行資料集的基準再提出新的特徵提取方式，並且優於現有基準的結果。 首先我們使用的是稀疏特徵提取方式，並且利用基於相機視角的弱監督學習方式，來找到密集的對應關係，但因弱監督學習方式無法明確計算的描述及檢測步驟的損失，所以這邊因此會導入基於相機視角的前處理，並且將描述步驟以及檢測步驟解耦訓練，採用先描述得到穩健且具判別力的描述子後再訓練檢測器，以取得更有鑑別力的描述點位置。 整體模型架構能分成以下三個部分，首先基於相機視角的前處理，以利描述網路計算損失，可以稱為線至窗口方法，主要利用兩相機內外參數及對極線轉換的概念，來協助優化描述網路的權重。 再者相機描述網路，這邊我們提出了新的架構，更專注於全局的視野感受，並導入了多感受域以及基於注意力的全局資訊補償資訊，在這些模塊的作用下一定程度的提高了整個網路的描述能力。 最後檢測網路部分則是以較簡單的網路架構作特徵點位置的提取，並利用強化學習來優化網路，其中的改進是我們在訓練網路時加入了特徵點位置的對應關係，這邊我們利用了兩影像的對應關係，此操作過程相同於特徵匹配方式下的估計基礎矩陣的方式，利用隨機抽樣一致性算法計算出兩張照片特徵點的對應關係矩陣並，再將估計矩陣與資料集提供的實際值計算兩者間的損失作為獎賞之一，並加上成對照片上取出特徵點的特徵相似度等資訊，完成最後的優化步驟。 我們的模型對比於現行已提出的弱監督模型，能在被廣泛使用的公共資料集上面獲取與其他模型可匹敵的結果，甚至對於照明改變影像的任務上能獲得優於在此前提出模型的結果。

With the development of computer vision, effectively describing image information has become an important issue. Feature extraction is one way used to describe images and is applied in various tasks such as object detection, object tracking, image stitching, 3D visual reconstruction, and localization. In the past, different feature extraction methods were proposed, including handcrafted methods based on image texture, edges, shapes, etc., utilizing gradients or mathematical relationships like derivatives for manual feature extraction. Recently, there is a popular trend towards deep learning-based feature descriptors, employing multi-layer convolutional neural networks or attention mechanisms to capture local and global image information effectively. These approaches leverage a substantial amount of data for training to optimize the neural network results. Both of these approaches are common ways of describing features. However, deep learning-based methods have recently shown excellent performance on many datasets. We will prompt the proposal of new feature extraction methods based on the benchmarks of existing datasets, outperforming current standards. First, we employ a sparse feature extraction method and utilize weakly-supervised learning based on the camera perspective to find dense correspondence relationships. However, due to the inability of weakly supervised learning to explicitly calculate the loss of the description and detection steps, we introduce camera-perspective-based preprocessing. In this process, we decouple the training of the description and detection steps, addressing the challenge of weakly supervised learning by adopting a two-step approach. We first train the description step to obtain robust and discriminative descriptors, and then we train the detector to achieve more discriminative feature point positions based on the obtained descriptors. The overall model architecture can be divided into three parts. Firstly, the camera-perspective-based preprocessing is introduced to facilitate the calculation of the loss in the description network, referred to as the line-to-window method. It primarily uses the concepts of camera intrinsic and extrinsic parameters and epipolar line transformation to help optimize the weights of the description network. Secondly, the camera description network is proposed with a focus on a global field of view and the incorporation of multi-scale receptive field and attention-based global information compensation modules, enhancing the overall network's descriptive capabilities to some extent. Finally, the detection network extracts feature point positions using a relatively simple network architecture and employs reinforcement learning for optimization. An improvement in this step is the inclusion of correspondence relationships between feature point positions during network training. We use the Random Sample Consensus (RANSAC) algorithm to get the fundamental matrix between two images, a process similar to estimating the fundamental matrix under feature matching, then compute the loss between the estimated matrix and the actual values provided by the dataset. This is complemented by information such as feature point similarity extracted from paired images, completing the final optimization step. Our model, compared to existing weakly supervised models, demonstrates comparable results on widely used public datasets. Moreover, it even outperforms previous models in illumination-changed image tasks.