利用影像分類技術識別網頁及手機程式中之圖形介面元件

Abstract

隨著網路與智慧型手機的普及，網頁及手機應用程式如雨後春筍般出現。若是一個應用程式在功能上或性能上有缺陷的情況下上架，它很快就會在市場中消逝，並造成開法者巨大的損失。因此，如何對網頁及手機應用程式進行有效的測試成為一個至關重要的議題。 當進行圖形介面程式的測試時，我們需要一個爬蟲來盡可能攫取圖形介面中的資訊，藉以設計相應的測試腳本。在圖形介面程式中，通常包含文字與圖像的部分。對於圖像的部分，我們難以單憑網路爬蟲解析其本質。然而，藉由圖像分類的技術，我們可以識別圖像所代表的意義。這麼一來，我們便能完整且準確的理解整個圖形介面的內容，以產生適合的測試腳本。 在這篇論文中，我們蒐集了31490張應用程式中常見的圖示，包含57種不同的種類，並利用卷積神經網路的技術在此資料集上訓練出圖像分類的模型。我們採用了適當的資料擴增方法，使得在預測現實世界程式中的的圖像時能達很高的準確度。此外，我們建立了一個可以持續更新模型的架構。

As the internet and smart phones become more and more common, plentiful web and mobile applications show up. If an application is published with even some tiny flaws in functionality or performance, it will fade away in the market rapidly, and the developers will suffer tremendous losses. Consequently, how to test web and mobile applications effectively and efficiently become an important issue. When conducting software testing for GUI applications, we need a crawler to grab the information of the GUI contents as much as possible in order to devise test scripts accordingly. It is general that there are both text contents and image contents in GUI applications. For the image contents, we may not always resolve the essence of them only by a crawler. However, we can recognize the meaning of image contents with the aid of image classification techniques. By doing so, we can understand the whole GUI contents thoroughly and accurately and generate suitable test scripts. In this thesis, we collect 31490 images of 57 different classes commonly seen in real applications and use CNN to train a model to classify the images. We adopt some appropriate methods of data augmentation to reach high accuracy of predicting image contents in real-world applications. Besides, we build a framework update our model continuously.