You only need to cache for tokens that are actually being used. Using 100k tokens of a 256k token window takes the amount of memory as using 100k tokens of a 1m token window.
which is why they charge extra for an option to hold >256K tokens.
The cost may be smoothly variable, but likely there's a bimodal distribution of users who barely use any context and users who push it to the max. Average price across both extremes fits nobody, but averages per kind of workload can be close enough.
Having it as a separate model makes it easier to load-balance the traffic.
I like to use Pi (https://pi.dev/), and I recently got it to make an approval extension for itself. It has a lot of documentation built-in for the agent to modify the behavior of the app.
I got it to display all proposed file change diffs and bash commands and made it so I can either approve the action or deny it with a message for it.
It was surprisingly easy to tell it to modify things things the diff viewing algorithm or syntax highlighting for the diffs.
haha i just downloaded pi after browsing HN for a bit. i'm really trying to get off the anthropic train before the subsidies explode and we're left holding onto codebases we don't understand and can't afford.
What models do you use for the diff level edits? My concerns would be speed, though I'm coming from letting opus ruminate for minutes at a time so my heuristics may be off.
I use a $200/mo OpenAI Codex sub. Throughout the workday I run an average of 2 concurrent agents of GPT 5.5 with high reasoning on fast mode and use less than half of my subscription usage.
For more interactive/active usage you might be better off using the low reasoning level, but I have usually found high to be a good balance of intelligence and generation speed.
There's an NPC who walks you to the pokemart and gives you a potion. For me that dialogue was something like "You receive a !" and it then said "You place it in the 23 pocket" -- seems related?
Thanks for the info, I will look into this! I got past that battle and further into the game in my testing, but I don't think I've tried using an item in battle yet.
