利用影像辨識方法之應用程式自動化測試技術

Abstract

圖形使用者介面是現今一種重要的軟體設計方法，它能讓使用者輕鬆的跟硬體互動，但是對與軟體測試工程師而言，如何界定可以操作的區域是一大挑戰，現今有兩種常用的作法，一種是讓測試工程師用人眼自己判斷，這種方法需要花費大量的人力，另一種是經過平台提供的應用接口來取得畫面的定義，這種方法可以快速又省力的取得定義，但是無法取得以外部元件設計出來的圖形使用者介面的資訊。 因此我們提出另一種方式，使用電腦視覺的技術去模擬人類分析當前看到的畫面，這種方式一樣能節省人力，又可以避免無法取得資訊的畫面。

Graphical User Interfaces (GUIs) are one of the most important parts of applications in nowadays. GUIs allow users to interact with hardware in user friendly ways, but are big challenges for software test engineers to examine the input interaction space of GUIs. There are two ways to examine the input interaction space. One is that test engineer find out them manually. This method cost lots of manpower. The other way is that using the platform API get the information automatically. This method save lots of times and human power but we could not get GUIs generated by external components with the platform API. Therefore we use computer vision technology to analysis the screenshot get the information as human. This method can also save manpower and avoid lost information of external components.