When you build a chat feature on an AI API, it is natural to estimate cost per message. That estimate will be too low, because AI model APIs don't remember earlier messages on their own. To keep the conversation going, your app sends the whole history again with every new message.
How input grows
Imagine a chat with a 1,000-token system prompt, where each user message is about 200 tokens and each reply about 300. On the first turn, you send 1,200 tokens. On the second, you send the system prompt, the first exchange and the new message: 1,700 tokens. Each turn adds another 500.
By the tenth message, a single request carries 5,700 input tokens, even though the new message is only 200 of them.
If nothing were resent
10 × 1,200equals12,000 tokens
Earlier turns resent
500 × (0 + 1 + 2 + … + 9)equals22,500 tokens
Actual input sent
12,000 + 22,500equals34,500 tokens
What it costs
At $2 per million input tokens and $10 per million output tokens, the 34,500 input tokens cost $0.069 and the 3,000 output tokens cost $0.03, for about $0.10 per conversation. If history were not resent it would be about $0.054. Across 1,000 conversations a month, that is roughly $99 instead of $54.
Because the resent history grows with every turn, doubling the length of conversations more than doubles their cost.
Ways to keep it under control
- Prompt caching: many providers charge less for input that repeats from one request to the next, such as a long system prompt.
- Summarising: replace older turns with a short summary once a conversation gets long.
- Trimming: drop the oldest messages beyond a certain length, if the task allows it.
- Shorter replies: output tokens are priced higher and also become input on the next turn.
Tip: When you estimate costs, model a realistic conversation length rather than a single message. Averages from real usage logs are best once you have them.