ACTIVIEW: EVALUATING ACTIVE PERCEPTION ABILITY FOR MULTIMODAL LARGE LANGUAGE MODELS

About ActiView

ActiView is a benchmark designed to evaluate the active perception abilities of Multimodal Large Language Models (MLLMs). In this task, models are challenged to navigate and adjust their filed of view to answer questions, simulating human-like active perception.

Authors

Ziyue Wang¹, Chi Chen¹, Fuwen Luo¹, Yurui Dong², Yuanchi Zhang¹, Yuzhuang Xu¹, Xiaolong Wang¹, Peng Li¹, Yang Liu¹

¹Tsinghua University, Beijing, China; ²Fudan University, Shanghai, China

Abstract

Active perception, a crucial human capability, involves setting a goal based on the current understanding of the environment and performing actions to achieve that goal. Despite significant efforts in evaluating Multimodal Large Language Models (MLLMs), active perception has been largely overlooked. To address this gap, we propose a novel benchmark named ActiView to evaluate active perception in MLLMs. Since comprehensively assessing active perception is challenging, we focus on a specialized form of Visual Question Answering (VQA) that eases the evaluation yet challenging for existing MLLMs. Given an image, we restrict the perceptual field of a model, requiring it to actively zoom or shift its perceptual field based on reasoning to answer the question successfully. We conduct extensive evaluation over 27 models, including proprietary and open-source models, and observe that the ability to read and comprehend multiple images simultaneously plays a significant role in enabling active perception. Results reveal a significant gap in the active perception capability of MLLMs, indicating that this area deserves more attention. We hope that our benchmark could help develop methods for MLLMs to understand multimodal inputs in more natural and holistic ways.

Project introduction

Despite the extensive efforts devoted to MLLM evaluation, active perception remains underexplored. Active perception involves understanding the reasons for sensing, choosing what to perceive, and determining the methods, timing, and locations for achieving that perception^[1]. This is important because in the real world, the desired information often does not appear directly in the center of one’s field of vision. Instead, it requires individuals to move their field of view, locate details, and filter out distracting information. For example, in right figure, suppose we are looking for information in a giant painting. We need to first shift our view to locate the specific area and then possibly zoom in to gather detailed information. Intuitively, active perception not only enables a person or model to accomplish more complex tasks, but it also has the potential to serve as a good indicator of the level of intelligence of a model. This makes it a critical capability that warrants thorough evaluation.

Benchmark

(Explore the code and datasets for ActiView: View on GitHub)

Our benchmark exams active perception abilities of models via different perceptual fields, Actively zooming and shifting of Views (ActiView) are required. We summarise zooming and shifting as two fundamental factors of active perception, where we can evaluate active perception abilities of models through the two factors separately or integratedly. ActiView imitates the procedure of active perception by providing models with an initial view, which is a cropped field of the original image, as depicted in the above figure, and requires models to search for missing but important information with view zooming and shifting, and to minimize distractions caused by redundant information in the view.

When perceiving an image, humans intuitively focus on three principle aspects: the environment depicted in the image, the primary objects, and the event that these objects are engaged in. Correspondingly, we summarise the questions in our benchmark into three main categorises, including environment-centric (Type I), object-centric (Type II), and event-centric (Type III) categorises. These main categorises are further divided into eight sub-classes according to the specific type of visual information and visual features used for answering the questions. The name of each sub-classes are shown in the above example and we refer readers to our paper for detailed explanations.

Pipeline

We design three pipelines for different abilities (as illustrated in the above figure), two fundamental independent abilities, zooming and shifting, and a mixture of them.

Zooming pipeline evaluates the ability to locate and determine fine-grained information that are necessary to answer the question (pipeline (a) above). It contains two stages, the view selection and the question answering stages. To simulate the zooming operation, models are required to first select sub-views to be zoomed given the initial view, then answer questions based on these zoomed views. The initial view used in this pipeline is the full image with size w × h. Each of the selected sub-views will be resized to size w × h, the same size as the initial view.

Shifting pipeline evaluates the ability to shift perceptual fields for missing information and to deduce the answer given perceived perceptual fields, which is also a two-stage pipeline (pipeline (b) above). To simulate the movement of human eyes, models are presented with an initial view, of size w × h, which is a cropped field from the original image, and are asked to determine if the current view or views are sufficient for answering the questions repeatedly until the model is able to answer the question or all the views are consumed.

Mixed pipeline does not specify the type of active perception ability required, while the above pipelines require to either zooming or shifting. As illustrated in pipeline (c), models must independently decide whether to zoom and/or shift to different perceptual fields. Unlike the zooming pipeline, where the model answers questions based on all selected views, in the mixed pipeline, a view would be discarded after zooming if the model recognizes it as irrelevant to the question. In comparison to the shifting pipeline, the mixed pipeline also provides access to the full image view in addition to cropped sub-views.

Cases

We demonstrate three cases for the three proposed pipelines, with results generated by GPT-4 models and Gemini-1.5 models. Case (a) stands for the zooming evaluation, where models successfully identify the view containing useful information and generate the correct result. Case (b) illustrates a failure in the shifting-R evaluation, where all the models continue shifting to new views until all views are used. Though including the correct views, the additional views severely distract the reasoning process, where three out of four employed models produce incorrect answers.

To explore how human-like mixed evaluation affects the visual reasoning process, we further examine this failure case using GPT-4o. As shown in case (c), GPT-4o first zooms into the "upper left" and "upper right" views, then discards the "upper right" view and shifts to the "lower left" one, which finally leads to the correct answer. Notably, in the final preserved views, distracting information (the highest price tag on papaya, "69") is screened out. This indicates that GPT-4o exhibits decent active perception abilities to move the field of view, locate details, and filter out distracting information.

Leaderboard

References

[1] Ruzena Bajcsy, Yiannis Aloimonos, and John K Tsotsos. Revisiting active perception. Autonomous Robots, 42:177–196, 2018.