Optimizing AI Agent Selection

A Performance and Cost Analysis Using a Single PDF Dataset

Abstract

This study presents a comparative performance analysis of various AI agents, evaluating them based on execution time, memory consumption, token usage, and estimated cost. Using a single PDF document, we tested both simple and complex queries across multiple agents, including the OpenAI Agent, ReAct Agent, LLM Compiler Agent, and LLM Chain-of-Abstraction Agent.

Key findings reveal that the OpenAI Agent excels in execution speed but has the highest memory consumption, while the ReAct Agent is the most cost-effective due to minimal token usage. The LLM Chain-of-Abstraction Agent generates the most detailed responses but is the slowest and most expensive. Meanwhile, the LLM Compiler Agent offers a balance between speed, cost, and quality.

These insights highlight the trade-offs between speed, efficiency, and cost when selecting an AI agent, providing valuable guidance for optimizing AI-driven applications.