分析手機用戶關鍵操作以及自動化手機測試資料集生成

Abstract

在行動裝置 GUI 測試中，現有的開源數據集往往由於數據過時和變換有限，缺乏進行有效測試所需的精確性和相關性。本文介紹了一種雙重媒介視覺語言模型（VLM）系統，用於生成能準確捕捉行動應用互動的高品質數據集。通過結合 Transformer 和 UNet 模型，我們的方法使用 Android 模擬器自動化數據集收集過程，結果在精確定位動作和螢幕變換數據方面表現出色。

我們對所提出的自動生成數據集進行了評估，發現其在標記精確性方面超過了 RICO 數據集（85％ 對 83％），並且變換幀變異數數顯著較低（0.73 對 11.8）。我們結合了 3D UNet 和 Transformer 架構的視覺文字模型，展現出較其他配置更高的準確性（73％），凸顯了整合文字和視覺信息對於行動 GUI 測試的重要性。

這項研究強調了針對現代應用介面開發量身定制的數據集的重要性，並展示了自動化數據集生成以應對行動應用迅速變化的景觀的需求。所提出的視覺文字模型在處理行動 GUI 測試的複雜性方面被證明是有效的，顯示了結合視覺和文字洞察以進行準確分析的潛力。

In mobile GUI testing, existing open-source datasets often lack the accuracy and relevance needed for effective testing due to outdated data and limited transformations. This paper introduces a novel dual-agent Vision-Language Model (VLM) system to generate a high-quality dataset that accurately captures mobile app interactions. On the other hand, by leveraging a combination of transformer and U-Net models, our approach surpasses previous work in precisely locating actions on mobile phone screen recordings.

We evaluated the proposed auto-generated dataset and found that it surpassed the RICO dataset animation class in labeling accuracy (85\\\\\\\\% vs. 83\\\\\\\\%) and achieved a significantly lower variance transformation frame count (0.73 vs. 11.8). Our vision-textual model, which combined 3D U-Net and transformer architectures, exhibited superior accuracy (73\\\\\\\\%) over other configurations, highlighting the importance of integrating both textual and visual information for mobile GUI testing.

This research underscores the importance of developing datasets tailored to modern app interfaces and demonstrates the need for automated dataset generation to address the rapidly changing landscape of mobile applications. The proposed vision-textual model proved effective in handling the complexity of mobile GUI testing, showing the potential of combining visual and textual insights for accurate analysis.