基於卷積神經網路與 U 型卷積網路之影像感測器的色彩特性化建模

Abstract

本研究以卷積神經網路 (convolutional neural network, CNN) 與 U 型卷積網路 (U-Net convolutional network, U-Net) 演算法為核心技術， 並根據 CIE ( 法語：Commission Internationale de l´Eclairage，國際照明委員會 ) 所推薦基於人眼色彩視覺為基礎的色度測定標準，進行色彩特性化建模 (colorimetric characterization modeling)，以實現彩色影像感測器 (color image sensor) 之高精度色彩特性化。 影像感測器之色彩特性化是一項艱鉅的任務。首先彩色影像感測器所感測的 RGB 訊號不能當作色彩度量 (colorimetry)，因為相同的圖像以不同的影像感測裝置量測所產生的 RGB 訊號差異很大，同樣的 RGB 感測訊號可能代表不同的顏色，因此 RGB 訊號不是 CIE 所規範色彩度量的標準 ( 例如 CIELAB 或 CIE XYZ)。所謂影像感測器之色彩特性化係透過演算法進行 RGB 與 CIELAB/CIE XYZ 的色彩空間轉換。過去的研究主要採用包含對照表內插模式 (LUT-interpolation model)、迴歸模式 (regression model) 與類神經網路模式 (artificial neural network model) 等方法，到目前為止，色彩特性化的技術經過評測結果：還無法達到接近分光光譜儀的測色水準，其中主要原因為色彩特性化係一非線性的複雜關係，因此，色彩特性化的演算法還有很大的進步空間。 基於 CNN 卷積神經網路為基礎，本研究試圖突破傳統 (3 x N) 多項式迴歸建模的度量精度。對於 CNN 色彩特性化技術的研究，我們透過影像感測器自動擷取 IT8.7/4 色彩導表，將 (3 x 8 x 8 ) 像素 ( 3 為 RGB 三顏色， (8 x 8) 為像素大小 ) 輸入 CNN 卷積神經網路，再映射由分光光譜儀量測所輸出的 CIELAB (3 x 1 x 1) 像素 ( 3 為 LAB 三顏色 , (1 x 1) 為像素 ) 數據，經過 5 次迭代的卷積神經網路學習，到第 5 次迭代卷積層已擴增為 8 幅 (3 x 8 x 32) 特徵圖 (feature map)，最後平面化 (flatten) 生成 6,144 筆色彩特徵向量輸入至倒傳遞神經網路(back-propagation neural network, BP NN)。 在色彩特性化平均色差值的評比：CNN 建模的 ΔE*ab 為 0.48 優於傳統 (3 x 11) 多項式迴歸建模的 ΔE*ab 為 3.03。 CNN 色彩特性化建模面臨的挑戰：CNN 訓練所需的電腦運算量龐大、訓練時間長、訓練的色彩數據不足、與可驗證的色彩數據過少等。為了克服上述瓶頸， 本研究藉由 U-Net 突破 CNN 訓練運算時間的問題：U-Net 只花了 1,000 波期 (epoch) 的學習週期而 CNN 需要耗費 100,000 波期的學習週期。透過 U-Net 學習可以解決 IT8.7/4 色彩導表數據不足的問題：U-Net 僅從 ISO 12640 (CIELAB/ SCID) 的六幅影像，再利用資料擴增 (data augmentation) 技術標註 32,027,200 色面：U-Net 驗證 ISO 12640 (CIELAB/SCID) 的兩幅 CIELAB 影像中 9,338,456 像素與 1,626,192 顏色；而 CNN 從 IT8.7/4 色彩導表中驗證 39,488 像素與 317 顏色。 本研究利用 CNN 與 U-Net 卷積網路所建構之創新色彩特性化方法，相較於傳統 (3 × N) 多項式迴歸建模的性能表現更勝一籌，經由研究結果驗證 CNN 卷積神經網路的平均色差值 ΔE*ab 為 0.48，而 U-Net 的平均色差值 ΔE*ab 為 0.52，二者皆優於傳統 (3 x 11) 多項式迴歸模式的 ΔE*ab 為 3.03。雖然 U-Net 建模的平均色差值的精準度略遜於 CNN 模型，但是 U-Net 建模的運算效率比 CNN 建模快約六倍，實驗透過配備 Nvidia GPU GTX 1080 Ti 的 PC，驗證一張 ISO 12640 (CIELAB/ SCID) 彩色影像之特性化模型，CNN 模型運算平均需要 5 秒，而 U-Net 模型運算平均需要 0.8 秒。本研究證實藉由 CNN 與 U-Net 所產出的色彩特性化建模演算法技術，可提升影像感測器裝置之色彩特性化更高的精度。

In this study, the colorimetric characterization of a color image sensor was developed and modeled using a convolutional neural network (CNN) and an U-Net convolutional network (U-Net), in according with the International Commission on Illumination (CIE), which is the recommended colorimetric measurement standard based on human eye color vision as the basis. Color image sensors can be incorporated into compact devices to detect the color of objects under a wide range of light sources and brightness. They should be colorimetrically characterized to realize the high-precision color characterization of the innovative color image sensor technology. However, the colorimetric characterization of image sensors is a difficult task. First, the red, green, and blue (RGB) signals are not colorimetrically captured by the color image sensor. They are device-dependent, which means that different sensing devices generate different spectrum responses for the same RGB signal. Therefore, the generated RGB signals do not adhere to the colorimetry standards regulated by CIE, such as CIELAB and CIE XYZ. The colorimetric characterization of an image sensor applies algorithms to transform RGB to CIELAB/CIE XYZ color space. The previously used methods for this purpose mainly include the look up table (LUT) interpolation model, regression model, and artificial neural network model. The colorimetric characterization technology evaluated thus far has not reached a colorimetric measurement level close to that of a spectrometer. This is mainly because the system used for colorimetric characterization is nonlinear and complex, and thus, needs to be considerably improved. To address this issue, this study aimed to apply a CNN colorimetric characterization algorithm, which is superior to the traditional 3 × N polynomial regression modeling. Here, the color patch of the IT8.7/4 target is automatically captured using the image sensor and converted into (3 × 8 × 8) pixels, where 3 represents the RGB colors and 8 × 8 is the pixel size input to the CNN. Then, the image is mapped to CIELAB (3 × 1 × 1) pixels, where 3 indicates the three Lab colors and 1 × 1 is the pixel size that represents the data measured and output by the spectrometer. After five iterations of CNN learning, the convolutional layer expands to eight feature maps, each of (3 × 8 × 32) pixels, and finally flattens to generate 6,144 color feature vectors, which are input to the back-propagation neural network (BP NN). The average color difference value ( ΔE*ab ) for CNN modeling is 0.48, which is better than that obtained for the traditional 3 × 11 polynomial regression modeling (3.03). CNN colorimetric characterization modeling faces the following challenges: CNN training requires considerable computation, long training time, so far training color data, and negligible verifiable color data. To overcome the aforementioned issues, this study used U-Net. First, U-Net requires only 1000 epochs of a learning cycle, whereas CNN requires 100 000 epochs. Thus, the issue of excessive training time required for CNN computation can be resolved, Second, U-Net can overcome the issue of insufficient data in the IT8.7/4 target: it uses the data augmentation technology to label 32 027 200 color patches over 256 × 256 pixels, resulting in 2098 billion pixels, including 6 885 222 colors for only six ISO 12640 (CIELAB/ SCID) images; by contrast, CNN can only randomly select 1000 colors from the 1617 color patches in the IT8.7/4 target, which amounts to only 1000 patch colors over 64000 pixels. Moreover, U-Net validates 9 338 456 pixels and 1 626 192 colors in two CIELAB images of ISO 12640 (CIELAB/SCID), whereas CNN validates 39 488 pixels and 617 colors from the IT8.7/4 target. The innovative colorimetric characterization methods used in this study to apply CNN or U-Net, which are superior to the traditional 3 × N polynomial regression modeling. The study results validate the CNN performance. The ΔE*ab values of CNN and U-Net are 0.48 and 0.52, respectively, both of which are superior to the values achieved by the traditional 3 × 11 polynomial regression model (3.03). Although the accuracy of the ΔE*ab value modeled by U-Net is slightly inferior to that modeled by CNN, the computation efficiency of U-Net is approximately six times faster than that of CNN. The experimental results are validated using a PC equipped with Nvidia GPU GTX 1080 Ti. Using ISO 12 640 (CIELAB/SCID) standard color images for the colorimetric characteristic model, the application of the CNN model requires an average of 5 s, whereas the U-Net model requires 0.8 s. Thus, the colorimetric characterization modeling algorithm technology implemented using CNN and U-Net exhibit improved colorimetric characterization accuracy of the image sensor device.