In this paper, we explore how different AI agents perform when handling PDF queries—both simple and complex. We compare their execution speed, memory use, token consumption, and cost, focusing on the OpenAI Agent, ReAct Agent, LLM Compiler Agent, and LLM Chain-of-Abstraction Agent.
ABSTRACT
This study presents a comparative performance analysis of various AI agents, evaluating them based on execution time, memory consumption, token usage, and estimated cost. Using a single PDF document, we tested both simple and complex queries across multiple agents, including the OpenAI Agent, ReAct Agent, LLM Compiler Agent, and LLM Chain-of-Abstraction Agent.
Key findings reveal that the OpenAI Agent excels in execution speed but has the highest memory consumption, while the ReAct Agent is the most cost-effective due to minimal token usage. The LLM Chain-of-Abstraction Agent generates the most detailed responses but is the slowest and most expensive. Meanwhile, the LLM Compiler Agent offers a balance between speed, cost, and quality.
These insights highlight the trade-offs between speed, efficiency, and cost when selecting an AI agent, providing valuable guidance for optimizing AI-driven applications.