WebMall - A Multi-Shop Benchmark for Evaluating Web Agents

This page presents the WebMall benchmark for evaluating the capability of Web agents to find and compare product offers from multiple e-shops. WebMall is the first simulated environment benchmark with tasks requiring navigation of multiple web shops to collect and aggregate information given various levels of user-query specificity, as well as performing actions such as adding items to carts and finalizing a purchase by checking out. The WebMall task set spans tasks like searching and comparing offers for specific products, adding offers to the shopping cart, completing the checkout procedure, product searches given vague user requirements as well as searches for compatible products or cheaper substitute products and full search to checkout workflows.

Given a user task, the agent is asked to visit four e-shops exposing heterogeneous product offers via heterogeneous HTML interfaces. WebMall differs from existing e-commerce benchmarks, such as WebShop, WebArena, or Mind2Web, by 1) requiring the agent to visit multiple e-shops, 2) featuring heterogeneous product offers from different real-world sources, and 3) its increased task complexity, mirroring comparison shopping customer journeys.

01-08-2025: Release of WebMall-Interfaces: MCP vs RAG vs NLWeb vs HTML - A Comparison of the Effectiveness and Efficiency of Different Agent Interfaces to the Web
29-07-2025: Version 1.0 released: Featuring additional product offers, new task category "End to End", and cleanup of tasks und solutions.
05-06-2025: Initial release: Version 0.7 released.

The following screencasts show an agent performing different tasks across the four shops.

The WebMall benchmark covers 91 tasks distributed over 11 task categories grouped into five task groups. 10 out of 11 task categories require the agents to visit four different webshops to find relevant product offers. 4 out of the 10 further require the comparison of product prices across shops in order to find the cheapest offer(s). The task categories are grouped into Specific Product Search, Vague Product Search, Cheapest Product Search, Action & Transaction and End-to-End.

Specific Product Search tasks represent overview searches where users know the exact product or can formulate specific requirements and want to find all fitting offers in all shops. This includes Find Specific Product tasks (locating all offers for a named product across all shops) and Products Fulfilling Specific Requirements tasks (searches with constraints on attributes such as display size without naming the specific product).

Vague Product Search tasks incorporate the uncertainty present in real-world shopping when users are not aware of the available options or lack domain expertise. The tasks encompass Products Satisfying Vague Requirements (requiring reasoning about vague descriptions to return relevant offers), Find Substitutes (suggesting cheaper alternatives when items are unavailable or unsatisfactory), and Find Compatible Products (reasoning over compatibility, e.g., finding compatible CPUs for a motherboard).

Cheapest Product Search tasks require agents to find the cheapest offer(s) rather than all fitting offers and include Find Cheapest Offer (examining all shops for the lowest price on a named product), Cheapest Offer with Specific Requirements (constraint-based search with price comparison), and Cheapest Offer with Vague Requirements (reasoning about vague descriptions while comparing prices).

Action & Transaction tasks include Add To Cart (adding offers for specific products to the cart) and Checkout tasks (adding an offer to the cart and proceeding through checkout, including providing shipping and billing details via HTML forms).

Finally, End-to-End tasks combine searching for the cheapest offer, adding it to the cart, and completing checkout into a single workflow.

Each task is defined by a specific instruction to the agent in string format as well as the expected answers (URLs of the correct product offers). The instruction to the agent consists of a general part which is the same for all tasks and a task specific part. The general part contains links to the four webshops as well as instructions on how to submit the final solution after completing a task. An example of a complete instruction string is found here.

The table below gives an overview of the 11 task categories and includes an example task from each category. A list containing all 91 tasks of the WebMall benchmark is provided here.

The WebMall benchmark asks agents to search for products in four distinct webshops which provide heterogeneous user interfaces. The webshops are implemented using the WordPress plugin WooCommerce and can be hosted via docker either locally or on a remote machine.

Screenshot Shop 1: E-Store Athletics

Screenshot Shop 2: TechTalk

Each shop contains heterogeneous product offers originating from a wide set of e-shops which annotate product offers within their pages using the schema.org vocabulary. The product offers were extracted from the October 2024 version of the CommonCrawl by the WebDataCommons project [Brinkmann2023].

The four WebMall shops contain a total of 4,421 product offers distributed across three main categories: PC Components, PC Peripherals, and Other Electronics. The distribution varies across shops to create diverse shopping environments for agent evaluation. The PC Components category includes internal computer parts such as CPUs, RAM, and motherboards. PC Peripherals covers external devices like monitors, keyboards, and external hard drives, while Other Electronics features consumer tech products such as gaming consoles, headphones, and smartwatches.

We conduct a series of baseline experiments using web agents implemented as part of the AgentLab library that accompanies BrowserGym [Chezelles2025]. We test 8 agent setups along the three dimensions (1) observation space (AX-Tree or Screenshots or AX-Tree+Screenshots), (2) availability of short-term memory, and (3) the used LLM (GPT4.1 or Claude Sonnet 4). The observation space of the agent is either just the AX-Tree or screenshot of the visited webpages or both the AX-Tree and screenshot of the currently visible page. In the screenshot, each element of the visible page is annotated with a number that corresponds to the AX-tree id of the element. If short-term memory is activated, the agent can note down information it deems relevant to remember at each step. An example of the full final message passed to the agent, which also contains an action history, for two experimental settings with the GPT4.1 model can be found here (AXTree only) and here (AXTree+Memory).

Agent effectiveness is measured in terms of task completion rate, precision, recall, and F1 score. The completion rate is calculated as the fraction of tasks for which the agent returns all relevant answers without adding any non-relevant ones. The completion rate for add-to-cart and checkout tasks follows the same logic. Only if all relevant offers are added to the cart or checked out, is the task marked as complete. In order to capture partial task completion, precision, recall, and F1 are useful metrics. These metrics are computed per task and are subsequently macro-averaged. Metrics for efficiency are the average number of steps taken by the agent to complete a task, the average number of input and output tokens used by the agent to complete a single task, as well as the average runtime and cost per task. The results are presented in the tables below.

The following scatter plot visualizes the trade-off between cost per task and completion rate for different agent configurations, averaged across all tasks. Each point represents an agent setup with different observation spaces (AX-Tree, AX-Tree + Memory, AX-Tree + Vision, Vision) and models (GPT-4.1, Claude Sonnet 4). The plot uses a logarithmic scale for cost to better show the range of values.

For running the benchmark, we assume a unix operating system in order to run Docker. If you are using Windows, please refer to WSL setup of docker.

How to Setup the Shops

The setup consists of different Docker containers, two for each shop (shop + database). This allows for a simple setup using Docker Compose. Please refer to the installation guide on GitHub for setting up the shops.

Performing Experiments

Instructions on how to perform experiments on single tasks as well as running a full study on the benchmark sets is found in the GitHub installation guide. The system writes comprehensive logs during execution, including agent actions, observations, and performance metrics. All logs including summary results are stored in the output directory set in the .env file for later analysis.

[Yehudai2025] survey benchmarks for the evaluation of LLM agents and categorize them according to the agent's application domain as well as the agent capabilities that are evaluated. Other benchmarks that also evaluate the capability of Web agents to perform online shopping are the WebShop benchmark as well as the WebArena and Mind2Web benchmarks, which feature e-commerce tasks as part of a wider task set. Compared to these benchmarks, the WebMall benchmark requires agents to perform longer running tasks (due to visiting multiple shops), to deal with heterogeneous product data originating from different real-world sources, and perform advanced searches such as finding compatible or substitute products. A shopping benchmark that requires agents to perform product searches including conditions on user ratings and shipping details is DeepShop. DeepShop requires agents to perform searches on the live Web which limits the reproducibility of experimental results due to the evolving nature of the Web. In contrast, WebMall provides a simulated environment which allows the comparison of agents using exactly the same experimental setup.

We welcome feedback and contributions via GitHub issues and discussions. Alternatively, you can also contact the authors of the benchmark directly via email.

[Brinkmann2023] Brinkmann, Alexander, et al.: The Web Data Commons Schema.org Data Set Series. Companion Proceedings of the ACM Web Conference, 2023.

[Chezelles2025] Le Sellier De Chezelles, Thibault, et al.: The BrowserGym Ecosystem for Web Agent Research. arXiv:2412.05467, 2025.

[Yehudai2025] Yehudai, Asaf , et al.: Survey on Evaluation of LLM-based Agents, arXiv:2503.16416, 2025.

[Zhou2023] Zhou, Shuyan, et al.: Webarena: A realistic web environment for building autonomous agents. arXiv:2307.13854, 2023.

[Lyu2025] Lyu, Yougang, et al.: DeepShop: A Benchmark for Deep Research Shopping Agents. arXiv:2506.02839, 2025.