利用輕量模型進行位置感知視覺問題生成

Abstract

本工作引入了一項新的任務，即位置感知視覺問題生成（LocaVQG），旨在從與特定地理位置相關的數據中生成引人入勝的問題。具體而言，我們使用周圍的圖像和GPS坐標來表示這種位置感知信息。為了應對這個任務，我們提出了一個數據集生成流程，利用GPT-4來生成多樣且複雜的問題。然後，我們旨在學習一個輕量級模型，可以應用於邊緣設備，如手機。為此，我們提出了一種可靠地從位置感知信息生成引人入勝問題的方法。我們提出的方法在人工評估（例如參與度，連接性，連貫性）和自動評估指標（例如BERTScore，ROUGE-2）方面優於基線方法。此外，我們進行了大量消融研究，以證明我們提出的數據集生成和解決該任務的技術的有效性。

This work introduces a novel task, location-aware visual question generation (LocaVQG), which aims to generate engaging questions from data relevant to a particular geographical location. Specifically, we represent such location-aware information with surrounding images and a GPS coordinate. To tackle this task, we present a dataset generation pipeline that leverages GPT-4 to produce diverse and sophisticated questions. Then, we aim to learn a lightweight model that can address the LocaVQG task and fit on an edge device, such as a mobile phone. To this end, we propose a method which can reliably generate engaging questions from location-aware information. Our proposed method outperforms baselines regarding human evaluation (\eg engagement, grounding, coherence) and automatic evaluation metrics (\eg BERTScore, ROUGE-2). Moreover, we conduct extensive ablation studies to justify our proposed techniques for both generating the dataset and solving the task.