Today I had a short talk with one of my VC friends. While talking about some companies, she appreciated the 80% gross margin of software companies. My reaction was “well, you have to add 10% for OpenAI nowadays”.

I came back home and I reflected on it. COGS are about 20% (ofc, depending on the company). Well, hosting is 6-12% of revenues (so that’s up to 60% of the total COGS; source) and support or some other services make up the rest up to 20%.

According to Sarah Hinkuss from BCV (source), AI compute is about 1-3x the hosting costs. That leads to 6-36% of Revenues. So I was in the range, but rather on the low end of it.

This shows that AI software / apps have lower margins (potentially drastically lower) than non-AI counterparts (this sparked some debate recently, so this probably comes as no news to some of you). One can of course argue that you need it in order to stay competitive (just like cloud).

But then, I realized:

1. LLMs are competing heavily on price

And here’s the evidence I got - when Mistral AI released their new model at $2 per 1 million tokens, Together.ai, Abacus AI and Deep Infra subsequently followed and slashed the pricing down, getting it down to as low as $0.24 per 1 milion of tokens. That’s an 88% price drop! What’s crazier, it all happened within a few days. With time, prices will probably go lower.

Here’s how this impacts the gross margin:

compiled by Sarah Hinkuss - source

The above does not account for other potential lines in the COGS, but it gives a clear picture of how the margin is eroded less than initially expected. 10x efficiency in price!

Combining the findings from EY and BCV, we can infer that we could see AI only adding 0.6-3.6% to COGS with time (10% of the 6-36% above).

2. While AI adds a new line in COGS, it actually improves other lines in the P&L

And that makes a lot of sense. The AI is meant to optimize. This means cutting support costs and improving productivity. See below:

source: Tomasz Tunguz blog

Just look at Klarna. They are cutting off people because AI can do lots of their jobs. It could be just some fancy alibi as well (as many companies have done layoffs without blaming it on AI), but there’s probably some seed of truth - they probably found some ways to optimize with AI.

Hence, while COGS might be increasing, the other costs flowing towards EBIT / profits should be decreasing. In this case, is the gross margin losing some of it relevancy in the short term? I think it will be the case that we’ll have to look more at the bottom line. But overall, the gross margin should still signal whether you’re building software or not.

But here’s my takeaway for today - AI seems not to be that costly.