使用自然語言處理與圖像分類技術預測圖形介面元件主題

Abstract

為了確保網頁和行動裝置應用程式的品質，使用軟體測試技術來驗證系統是否有效。在自動軟體測試中，如果可以識別其中的元件會很有幫助。手動分析元件主題是一項耗時的工作。在之前的論文中，有通過文字標籤來識別元件主題的方法。我們提出了一種預測GUI元件主題的技術，它將圖像分類技術融入自然語言處理方法中。我們可以使用這項技術同時使用文字以及圖像屬性識別元件主題，並非只依賴文字屬性。只要將對網頁和行動裝置的所有元件進行爬蟲，並將其屬性記錄到文件中，便可使用該技術來判斷元件的主題。我們用python實作了這項技術並且作實驗。實驗表明，有著圖像分類技術輔助時，可以比原本的自然語言處理方法有更高的準確率。

For ensuring the quality of both web and mobile applications, software testing techniques are used to trace and observe whether the system is valid or not. In automatic software testing, it is helpful if the elements can be identified. Analyzing topic of elements manually is a time-consuming work. In the previous work, the topics of elements can be identified by the text tags. This paper presents a technique to predict the topics of GUI elements, which incorporate the image classification technology into the natural language processing method. With this technique, we can identify the topic of an element by its attributes, not only the text but also the image. All the elements of a web and mobile application are crawled and their attributes are recorded to a source file. Then we use the technology to judge the topics of elements. We implemented the technology with python and experiments showed that natural language processing approach can be improved when integrated with image classification.