Bytes and nibbles

To get a rough idea of inference costs, I decided to run my agent with a few different stocks and measure the cost.

Batch processing on LLM APIs is a specific service tier where your request is not serviced immediately, but is instead queued to be processed at some time (usually up to a day later) when the server has spare compute available. Batch processing saves a lot of money, but unfortunately, it doesn’t really work for agentic mutli-turn conversations due to the massive up to 24hr time delay between turns, so I decided to use standard tier (on-demand processing) and flex tier (a tier unique to OpenAI that is on-demand processing, but with higher latency than their standard tier).

To confirm whether this was simply random chance, I decided to run this test again with FDS, a value stock, but this time, caching was turned on for all Anthropic models and I ran FDS with perplexity enabled but the built-in web search disabled and with perplexity disabled but the built-in web search enabled. I wanted to see whether it was cheaper to use Perplexity or the built-in web search.

Perplexity costs $5 per 1000 requests + model input/output token costs, but the token costs are so negligible that I did not include them in my comparison.

GPT 5.4 is missing from the web search comparison tables because GPT 5.4 came out after I ran this test and when I tried running GPT 5.4 with web search, I kept getting rate limit errors. I am not sure why I got the rate limit errors for GPT 5.4, but not GPT 5.2 or GPT 5.1, but I didn't have time to debug it.

An interesting observation that can be made from this data is that the cheaper models like GPT 5.1, GPT 5.2, and Gemini 3 pro preview saved money by doing the web search themselves, whereas the more expensive models like Gemini 3.1 pro preview and the Claude models were cheaper when they offloaded the web searching to Perplexity.

I reviewed the logs and interestingly, I caught GPT 5.2 using the built-in web search to search ‘calculator: 201.71*37615000.’ I won’t implement it in this blog, but I clearly need to build a tool to allow my agents to calculate things. It's always worth reading the logs to ensure your agents are behaving correctly.

Key observations:

• Perplexity is cheaper than using web search for every model apart from Gemini 3 pro preview, GPT 5.1, and GPT 5.2.
• The cheapest models were: Claude Sonnet 4.5 (with Perplexity), Claude Sonnet 4.6 (with Perplexity), Gemini 3 Pro-Preview (with Perplexity), Gemini 3 Pro-Preview (with built-in web search), Gemini 3.1 Pro-Preview (with Perplexity), GPT 5.1 (with Perplexity), and GPT 5.1 (with built-in web search).

I then compared the cost of all of the models across all three stocks. For each of the AMZN Claude model costs, I used the cost when using message caching. For each of the FDS costs, I used the cost when using Perplexity. Additionally, for the rank on the Vals AI benchmarks, I ranked them while ignoring non-listed models

Interestingly, Claude Sonnet 4.5, Gemini 3.1 pro-preview, Gemini 3 pro-preview, and GPT 5.1 made very few tool calls compared to the other models. The available tools were the Perplexity web search and SEC filings search tool, so I was concerned that these models were not thorough enough in their research.

My sample of 3 test cases was clearly not a very robust sample size, so I decided to use Vals AI’s finance Agent v1.1 benchmark to inform my model choice as it evaluates the same task that the discount analyst currently handles.

Benchmark methodology (from the benchmark page):

The finance industry comprises a wide array of tasks, but through consultation with experts at banks and hedge funds, we identified one core task shared across nearly all financial analyst workflows: performing research on the SEC filings of public companies. This task —while time-consuming— is foundational to activities such as equity research, credit analysis, and investment due diligence. We collaborated with industry experts to define a question taxonomy, write and review 537 benchmark questions.

The AI agents were evaluated in an environment where they had access to tools sufficient to produce an accurate response. This included an EDGAR search interface via the SEC_API, Google search, a document parser (ParseHTML) for loading and chunking large filings, and a retrieval tool (RetrieveInformation) that enabled targeted questioning over extracted text. The human experts did not make use of any additional tools when writing and answering their questions. See the full harness here.

Our primary evaluation metric was final answer accuracy (see the GAIA benchmark). We also recorded latency, tool utilization patterns, and associated computational cost to provide a fuller picture of agent efficiency and practical viability. Together, these components form a rigorous and domain-specific evaluation framework for agentic performance in finance, advancing the field’s ability to measure and rely on AI in high-stakes settings.

Vals.ai Finance Agent v1.1 benchmark leaderboard (last updated 17th March 2026):

Vals AI also state that 'The dataset is divided into three parts: Public Validation (50 open-source samples), Private Validation (150 samples available for license), and Test (337 samples).'

To compare the models, I also decided to also add the overall score from https://www.vals.ai/benchmarks/vals_index to measure the general intelligence of the models. While knowing a lot about finance is helpful, having a lot of general knowledge is equally important to ensure that the model understands and can reason about the nuances of the stocks it is researching.

I plotted these data points with a Pareto frontier, the frontier of a multi-objective optimisation problem where at every point, a tradeoff must be made in one objective to achieve a better result in another objective. For example, Gemini 3 pro preview is Pareto optimal for the Vals Index because choosing a cheaper model yields a worse accuracy and choosing a more accurate model result yields a higher cost. On the other hand, it is not Pareto optimal in the Finance Agent v1.1 evaluation set as GPT 5.1 achieves a higher accuracy with a lower cost (i.e., there is no tradeoff between objectives).

Some clear takeaways from the graph are that Claude Sonnet 4.5, Claude Opus 4.5, GPT 5.2, and GPT 5.4 do not fall on the Pareto frontier of either evaluation set, so they should not be chosen.

One thing that this graph misses is that OpenAI offers a flex tier API, which offers slightly slower responses for half the price. I assume the Vals AI researchers used the full-price API, so introducing the half-price API changes the results significantly.

GPT 5.1 is still on both Pareto frontiers because it’s the cheapest model, but GPT 5.2 and GPT 5.4 entered the Pareto frontier because they became cheaper.

I decided to use GPT 5.1 as it was the cheapest ‘good enough’ model. To get a significant improvement in accuracy over GPT 5.1, a model that is 2-3x more expensive like GPT 5.2 (flex tier) or GPT 5.4 (flex tier) would need to used.

Sonnet 4.6 would provide a massive increase in accuracy, being best in class overall and in the Finance Agent benchmark, but I think I can make far greater gains in accuracy for the same price by enhancing my stock-picking workflow instead.

In my 3-stock sample, I noticed that GPT 5.1 used significantly less tool calls than the other models, which concerned me. Fortunately, the Vals AI Finance Agent benchmark shows it on average using a number of tool calls similar to other top-performing models, so this shouldn’t be an issue.

As I alluded to earlier in this blog, I plan to improve accuracy of my stock-picking agent by enhancing its workflow.

DCF analysis is incredibly sensitive to small changes in its parameters, so I decided to add a final step where for each stock analysed, I give an AI the deep research report and the results of the DCF analysis, and ask it to categorise the stock into: STRONG BUY, BUY, HOLD, SELL, or STRONG SELL.

In general, as you might expect, stocks with low values given by the DCF analysis tended to be given SELL ratings and stocks with high values given by the DCF analysis tended to be given BUY ratings. Interestingly, this did not hold true for all stocks.

Despite Microsoft being reported as overvalued according to the DCF analysis, the rating AI thought that ‘[the DCF analysis] is built on demonstrably conservative assumptions,’ and decided to RATE it as a BUY.

For PayPal, the rating AI argued that the company was even more undervalued than the DCF analysis suggested as the DCF analysis hadn’t taken into account several factor. The rating AI labelled this a STRONG BUY despite only giving a BUY rating to other stocks that had even higher margins of safety.