To be clear, they didn’t do anything wrong. The service is set up to let them do it. They had a pile of small work and a working endpoint, so they wrote some sort of loop. I probably would have done the same. But a loop that sends hundreds of thousands of tiny prompts is a batch job, and this was a really good reminder of why batch clusters are still super important.

What that load does to a shared service

A shared LLM endpoint is set up for lots of people asking questions here and there, interactive use. One person sending 800,000 of them back to back is a pretty different workload. Spread over a day, that’s roughly nine requests per second, all day and all night.

Every one of those requests costs something before the model even sees it. Each is a network round trip with its own connection overhead and its own spot in the queue. It also messes with caching. Inference servers lean on caches to keep things quick for the people already using them, and a steady flood of unrelated prompts pushes that useful state out. Time-to-first-response goes up for everyone, including the person asking a single question in a chat window who did nothing different.

Why a batch job is faster

On a GPU node in a batch job, you get the whole device to yourself. Nobody else’s traffic is in the queue, and there’s no network between your data and the model.

The bigger win is how the work gets handed over. Offline inference with a tool like vLLM takes the entire list of prompts at once and packs them onto the GPU as tightly as it can. Small prompts are where that pays off most, because one small prompt on its own leaves most of the GPU idle. Hand it 800,000 together and it keeps the device busy the whole time. You load the model once, point it at your dataset, and collect the results when the job ends.

If this is you

If your workflow loops over a dataset and calls an LLM once per row, please don’t forget about classic HPC. Write it as a batch job, run it on a GPU node, and keep the shared service for the interactive stuff it’s good at.

You’ll get more done quicker, and everyone else gets their response times back!

Related posts

• Serial Farms: Package Options and When to Switch to Farming

To be clear, there are actually two limits here, and they do pretty different jobs. The five-hour window controls when you can use your plan. The weekly cap controls how much you get in total. Most people worry about the five-hour one because it’s the wall they keep hitting, but in my opinion the weekly cap is the real budget.

How the windows work

On Claude Pro and Max, the five-hour session starts with your first message. It’s a rolling window, not a fixed clock time, and it resets five hours after that first message. The weekly limit is separate. It resets at a fixed time for your account, and waiting out a session won’t clear it. Chat, the desktop and mobile apps, and Claude Code all draw from the same pool.

Codex is split by plan. Plus and Standard Business have a five-hour window. Pro currently has none. Weekly limits “may also apply,” and OpenAI’s docs send you to your usage dashboard for the actual reset times rather than spelling them out.

Neither provider really counts messages. Model choice, context length, reasoning effort, and tool use all change how fast you burn through a window. A long agent session in a big repo with a top model eats through it way faster than a handful of chat questions for each of them though.

These rules change a lot, so check your own numbers instead of trusting a blog post, this one included. In Claude Code, /usage shows your session and weekly bars. In Codex, /status shows both percentages and when they reset.

Start the clock before you sit down

The five-hour clock starts on your first message, so you get to pick when it starts. If something sends a message at 7am, your first window closes at noon and you get a fresh one right after lunch. Now the morning is one window and the afternoon is another, instead of one window running right through the middle of your day.

The simplest version is pretty much just a ping. Most agent tools have a headless mode, so a cron job can send a throwaway prompt to a cheap model:

1

0 7 * * 1-5 claude -p "Reply with OK" --model haiku

codex exec "Reply with OK" does the same for Codex. Cron runs with a bare environment, so you may need the full path to the binary. A local cron job also only fires if your machine is on at 7am.

The nicer version is to have that early job do something useful. Most of the modern tools can run scheduled tasks, so set up a custom news brief. Have it cover what changed in the repos you care about, release notes for your tools, or whatever you’d otherwise skim over coffee. It starts the clock as a side effect, and you have something to read when you sit down.

The catch

This doesn’t ever raise your weekly cap. It gives you a second full window during your working hours, so you can burn through that cap faster. If you were already finishing most weeks with headroom, that’s a good trade. If you regularly hit the weekly wall on Thursday, you’ll hit it on Wednesday instead. Be careful.

What about peak hours?

For a while earlier this year, the time of day you used Claude mattered too. Weekday mornings from 5 to 11am Pacific drained five-hour sessions faster than the rest of the day. Anthropic removed that reduction for Claude Code on Pro and Max in May, and doubled Claude Code’s five-hour limits at the same time. Weekly limits didn’t double.

So off-peak usage isn’t worth more on Claude Code right now. Providers have done it before, though, and could easily do it again, so check whether it applies to your plan before you plan your day around it.

What I’d do

If the afternoon wall is what keeps getting you, try the early ping or a morning brief. Just keep an eye on that weekly bar!

Related posts

• Why I still use Sol on low

In my opinion, Tectonic and a coding agent are now a pretty practical alternative to Overleaf if you’re working on a document by yourself. The agent can work directly on local files and run the compiler. This way you don’t need it clicking around a browser through computer-use tools trying to diagnose how your tables are formatted.

A simpler local setup

If you haven’t used LaTeX, it’s a way to write documents as source files and compile them into typeset output, usually a PDF. It’s useful for résumés and papers, especially when you want precise formatting or have lots of equations. Your textbooks use it, for example.

Getting a local LaTeX setup working can be more effort than you wanted to spend on editing a document. Some of this technology is unfortunately pretty ancient. Tectonic makes that part simpler. It’s a TeX/LaTeX engine distributed as a single executable, and it downloads the support files it needs instead of requiring you to install a full LaTeX distribution up front.

There are installation options for different systems. Once it’s installed, compiling a document can be as simple as:

1

tectonic resume.tex

That produces resume.pdf. Tectonic also handles the repeated compilation passes that TeX and BibTeX can need. You’ll need network access to download support files that aren’t already cached, but you don’t have to manage those downloads yourself.

Let the agent keep trying

The useful part for me was making the changes interactively. LaTeX has plenty of classic formatting problems. Something doesn’t fit, spacing looks wrong, or a change in one place moves something somewhere else. An agent doesn’t really run out of things to try, which is pretty useful when you’re working through that kind of problem.

Ask the agent to edit the source, compile it, and then render the PDF pages to images for visual inspection if it has the tools to do that. If the change didn’t work, it can keep trying. Tectonic makes it easy for the agent to build/compile the PDF, and then other standard agent tooling can view the PDF for issues.

That last step is the big win. A successful compile tells you that you produced a PDF, not that the layout looks right. Ask the agent to review the rendered pages rather than stop as soon as the command succeeds. You should still look at the final document yourself every time of course.

This doesn’t replace Overleaf’s shared editing and collaboration features. But for a document I’m working on myself, letting an agent edit and compile the files directly is a much simpler workflow than having it operate a web editor.

If you’ve got an existing LaTeX document, try giving your agent the source and asking it to make a change with Tectonic. It worked pretty well for my résumé!

Say I have a ticket to implement an entire feature branch. The agent has to read the existing code, decide what needs changing, deal with whatever it finds along the way, and test the result. With Sonnet, I’d expect to need to help break that into smaller jobs. I’d essentially be pair programming with Sonnet, with it controlling the keyboard. This is totally fine, but missing a lot of the strengths of the frontier models.

On the other hand, Opus is the one I’d try giving the whole ticket to. No single step has to be especially hard. The agent is just doing more on its own without steering. Maybe it finds that the existing tests don’t cover the change, or that the first implementation needs work once it runs them. With Sonnet I’d expect to stop, look at that, and decide what to ask for next. With Opus I’d give it room to investigate and keep going. That’s the difference I care about when I say “wide,” not whether Opus writes a better line of code.

This isn’t a claim that Sonnet can’t implement the feature. If I’m there to break down the work and steer it, I’d expect it to do fine in the average case. Models like Sonnet and Luna have their place, but I wouldn’t choose them for a job where I want to leave the agent alone to work through all those branches.

Of course, I still read the changes, run the tests, and check the result against the ticket. If the ticket involves decisions I need to own, I don’t hand those over either. I just don’t want to have to tell the agent what to do next every five minutes.

If you’ve only been giving agents small jobs, try giving Opus a whole ticket and see how far it gets. Then check its work.

Related posts

• Why I still use Sol on low

Consider an ERP study and two cleaning policies. One rejects nearly every blink, along with a lot of trials. The other leaves a little ocular activity but keeps more trials. If I only count blinks, the first one wins. If I only count retained trials, the second one wins. Neither one is really better than the other.

I’d be wary of picking a method because it has a higher signal-to-noise ratio or fewer outliers despite how many publications in the field report this way. Those numbers depend on how you define signal and noise. My old metric counted the difference between raw and cleaned data as noise. By that definition, removing more data looked like progress. A flat recording could also score well on a variability metric and be useless for the study.

That doesn’t make artifact measures pointless. I just want to know whether a blink correction reduced blink contamination. I also want to know what it cost.

Compare what you get and what you lose

For an actual study, I’d put a few plausible preprocessing choices next to each other and ask:

• How much targeted artifact remains? Which type of artifact hurts us the most?
• How many channels, time periods, and trials did each choice remove or reconstruct? Did that differ between conditions or participants?
• What happened to the effect estimate and its uncertainty, such as its confidence interval?

A narrow confidence interval isn’t automatically good. If a pipeline systematically distorts the effect, it can give a precise answer to the wrong question. A smaller p-value isn’t always the goal either. If two pipelines disagree, I’d look at what each one removed before picking the result I like better.

Real recordings usually don’t tell us the true effect, so comparing their results can’t establish which pipeline is correct. If I can simulate data or inject a known effect into realistic recordings, I can ask a harder question: does the method recover the effect without systematic bias? Across repeated simulated datasets, do nominal 95% confidence intervals contain the known effect about 95% of the time? How wide are those intervals, and how much data did the method discard to get them? This is the sort of thing I want to see in pipeline style publications.

The answer might depend on the study. A method that throws away a lot of time could be acceptable for one analysis and disastrous for another. An aggressive trial-rejection rule might remove artifacts while leaving too few trials for a useful estimate, or reject one condition more often than the other. That’s information a single cleanliness score won’t give me.

I don’t expect every EEG study to run a giant pipeline benchmark or accidentally turn into a methods paper. A useful first pass is to run a couple of defensible cleaning choices on the same data and put the artifact measurements, data-loss counts, and effect estimates side by side. If we can test against a known effect, we should check whether the method recovers it. That’s more work than picking the cleanest-looking trace, but at least it tells us what we paid for the cleaning.

Related posts

• How Do You Test Whether EEG Got Cleaner?

It also overengineered some open-ended problems I gave it. And I can’t use it every day without upgrading my plan. That matters more to my recommendation than how impressive it was.

What I actually use

GPT 6 Sol on low is my default in Pi. So far it feels like a flat upgrade over 5.6 Sol for my work. The change I notice most is the prose it writes without any writing skills. Low is enough for most tasks. I only turn up the reasoning for really hard ones.

Fable is amazing for code I can trust enough to review and merge. I can give it a wildly broad task, and if it gets stuck, it comes back to me instead of burning tokens forever. I don’t have it on my current plan, though. Sol on low does enough that I rarely need to reach for the big guns.

Opus 5.5 is what I’ve started using for my longest-running tasks with lots of context. It feels fast, and the prose is way better. The “Claude speak” that bothered me before seems to be gone. Opus 5 was a flat no for me. 4.8 was alright, but Sol 5.6 beat it easily. I didn’t expect to come back to Opus, but here I am.

The plan changes the answer

My recommendations assume subsidized subscription plans, not pay-as-you-go API pricing. Running agents at API rates every day would be too expensive for me. Check what’s included in your plan before copying anyone’s model ranking. The expensive option needs to do something your cheaper default can’t.

I’ve updated my Recommended AI Tools page with what I use now. If you’re choosing a setup, start with a model you can afford to run often, then work out which tasks actually need something bigger.

Related posts

• When to reach for Opus
• Check what your AI plan does with your data
• Stop installing things your agent doesn’t need
• Start your five-hour window early

How I choose

Typically, I open up Pi and coordinate my tasks based on GitHub issues via the Matt Pocock skills. GPT 6 Sol on low is my daily driver. I only raise the reasoning level for really hard work. For long-running tasks with lots of context, I’ve started reaching for Opus 5.5.

These recommendations assume you’re using a subsidized subscription plan. Paying API rates for all your day-to-day agent work gets too expensive. Check what’s actually included in your plan before making an expensive model your default.

Harnesses

1. Pi: My daily driver. It ships as a bare skeleton without extensions. The cool part is that its own documentation and code are available in its context, so it can modify itself. Say you want a modern “goal” or “btw” feature like the ones in Codex or Claude Code. You ask Pi to build it, and it will. Pi is really meant for enthusiasts who want aggressive control over everything. These are the extensions I use:

   • add dir: Symlinks other directories into your context.
   • web search: Self-explanatory.
   • power bar: Shows token use, context stats, and more in a toolbar below the prompt line.
   • ask user question: A rip-off of Claude’s questions feature.
   • goal light: My own super-lightweight goal and loop framework.
   • prompt stats: My own summary of the current system prompt and tools.

2. T3 Code: More of an orchestration platform. It lets you combine all your subscriptions and get away from the terminal, making it easier to copy and paste screenshots, text blocks, and so on. It also has very cool remote access support. You can start a session from your phone and run it across a bunch of different machines. Claude and Codex have similar features, but T3 is better at working with a distributed fleet of machines.

   • It is open source, so you can change things pretty easily.
   • As soon as they add Pi support, I would daily-drive this.

3. Codex: If you already have a ChatGPT subscription, start here. It has the best “computer use” skills on the market right now. You can have it control Chrome or other apps on your computer to find buried settings and handle similar tasks. I don’t use it much anymore unless I need to fill out a ton of online forms or do something like that.

4. Cursor: This was my entry point into the space. It started as a VS Code fork with advanced AI features, though it does lots of other things now. I appreciate that Cursor customizes its prompts and harness settings for every model you select. It is also a good starting point.

5. Claude Code: The Claude models are amazing, but both the desktop app and the TUI feel a bit dated by my standards. Many of the killer features they pioneered now exist in the tools above, along with more interesting additions.

Models

I don’t think a strict ranking captures how I use these. Here’s what I reach for and why:

• GPT 6 Sol: My daily driver in Pi on low. So far it feels like a straightforward upgrade over GPT 5.6 Sol, including better default prose without writing skills. Low is enough for most tasks. I only turn up the reasoning for really hard ones.
• Astra: The first model release that genuinely surprised me. Without being asked, it spun up multiple Playwright browsers to test and compare its work and recorded video to show it had finished. It can overengineer open-ended tasks, though, and it costs too much for me to use every day.
• Fable: Excellent at producing code I can trust and merge. If I give it a huge, nearly impossible task, it comes back to me when stuck instead of burning tokens forever. Plan access and cost keep it from being my default, especially when Sol on low is usually good enough.
• Opus 5.5: My choice for long-running tasks that need lots of context. It feels fast, and its prose is much better than earlier Opus releases. Opus 5 was a flat no for me, 4.8 was alright, but Sol 5.6 beat it easily.
• Cursor’s models: A good way to try a variety of models without building my setup. Cursor tunes the experience for its models, and I’d still suggest it as a starting point.

I haven’t used Luna 6 yet, so I can’t recommend it either way. Local models have their place, but they’re beyond the scope of this page.

Outside coding

I replaced a self-hosted Hermes setup for my curated daily news brief with Grok Bot. It did the job better for me, without requiring me to host something I considered potentially risky. That’s a recommendation for that particular job, not a coding-model ranking.

Skill packages

I use only two major skill sets. I use one almost in full and took what I wanted from the other. I also create one-offs for certain projects when it makes sense, but those aren’t really shareable.

1. Matt Pocock’s Skills

   • A great way to hand the project management side over to agents.
   • Grilling and Wayfinder are amazing for exploring ideas and putting them into issue trackers.
   • The instructional content and docs are really good.
   • Try adopting as much of this as you can, minus the obviously goofy skills, and give Wayfinder a shot.

2. pstack, aka Poteto Mode

   • The writing skills are awesome, especially Unslop.
   • It has more skills and workflows for working with parallel agents.

We used 10,000 GPU-hours.

It looks like a useful answer. There’s a number and a unit. Very official.

But what does a GPU-hour actually measure? On its own, pretty much nothing. I can’t compare that number with another project until I know what counted as a GPU, whether it was a whole device or a slice, and how the authors calculated the total.

One GPU-hour is not like another

An hour on an NVIDIA A100, H100, or B200 counts as one GPU-hour under the usual calculation. That doesn’t make those hours equivalent. These are different generations of hardware with different memory capacities, bandwidth, and compute capabilities. They have tons of different features too.

The workload matters too. A newer GPU might make a huge difference for one model and barely move another. number of GPUs × elapsed time throws all of that information away. It’s pretty much the worst collapsing across dimensions you could do.

In the past, research papers have long reported CPU models and architectures because an hour on one processor doesn’t automatically compare with an hour on another. Plenty of papers still include that detail, and they should. GPU reporting deserves at least as much care.

Device slicing makes it worse

Multi-Instance GPU, usually called MIG, is device slicing. An administrator can divide one physical NVIDIA GPU into isolated instances, each with a portion of the compute and memory resources. NVIDIA’s MIG documentation says supported GPUs can be divided into as many as seven instances.

This is a good thing. Several jobs can share an expensive GPU without fighting over the same resources. Schedulers can present each slice to a job much like a physical GPU. Slurm can schedule MIG instances as individual GPU resources, for example.

Here’s where the reporting gets weird. Say two jobs each run for one hour. One gets an entire H100. The other gets one MIG slice of an H100. If the accounting system calls each allocation one GPU, both jobs used one GPU-hour. Same number. Very different allocation.

Calling these “MIG-hours” or “slice-hours” doesn’t fix much by itself. MIG profiles assign different amounts of compute and memory. Ten hours on a small slice and ten hours on a large slice still become ten slice-hours when the report leaves out the profile.

Show your math

Different people need different numbers. A cluster operator may care about how long allocations were occupied. A funding report may care about use of the physical hardware. A researcher needs enough detail to understand or reproduce an experiment. Those are different questions. They don’t have to produce the same total.

One report might count every scheduled MIG instance as a GPU. Another might convert slices into fractions of a physical device. It could count allocated time even when the job wasn’t keeping the device busy. All of those approaches can make sense for a particular job. Calling every result “GPU-hours” is the problem. Once the calculation disappears, the reader can’t recover what happened.

A universal normalized GPU-hour sounds tempting, but I don’t think it fixes this. Converting every device and slice into one equivalent unit requires assumptions about relative performance. Those assumptions change with the workload, software, numerical precision, and bottleneck. It’s problematic either way.

At least report this much

If you publish GPU usage, include:

• the GPU manufacturer and model
• whether jobs received whole devices or slices
• the MIG profile or equivalent partition size when slicing was used
• the number of allocations
• the elapsed duration
• the exact calculation behind any aggregate total

A methods section or figure caption could say:

We used eight allocations of an NVIDIA H100 3g.40gb MIG profile for five hours each, for 40 H100 3g.40gb slice-hours. We calculated slice-hours as the number of allocated instances multiplied by allocation duration.

Yes, that’s longer than “40 GPU-hours.” You can also tell what it means. The reader knows these were slices, which hardware they came from, and how the total was produced.

If the work used several device models or MIG profiles, report them separately. Don’t add unlike allocations together and slap a precise-looking label on the result.

A caption is allowed to do some work

A single GPU-hour total fits nicely on a dashboard. The useful version takes a caption, a definition, or a few lines in a methods section. That’s fine. This is what those things are for.

GPU-hours can still work as internal shorthand for one stable pool of identical hardware. Everyone in that context knows what the unit means. The shorthand falls apart as soon as the number leaves that context.

Don’t make readers guess. Name the hardware, describe the allocation, and show your math.

Related posts

• Your KPIs Aren’t Telling You Enough

I’m not saying retrieval-augmented generation is useless. It can make a large collection of documents cheap and fast enough to query. But if you have some tokens to burn and your corpus isn’t massive, it may be better to let an agent search the original documents itself.

I ran into this with a public-facing technical wiki. The articles are dense. They have long tables, code samples, configuration details, and links to many related pages. A useful answer might depend on a warning above a table, one row in the table, and an example farther down the page.

Chunking pretty much destroys this on average as the length of documents grows.

Retrieval commits too early

A typical RAG pipeline splits documents into chunks, creates embeddings for them, and retrieves a small set that appears relevant to the question. The model answers using those chunks.

That works when each chunk still makes sense on its own. A paragraph from a straightforward policy document might survive the process just fine. Technical documentation is less cooperative.

A chunk can contain half a table without its heading. A code sample can lose the paragraph that explains when to use it. Two adjacent chunks can look unrelated to an embedding search even though a human reading the page would clearly use them together.

You can improve the pipeline. You can change the chunk size, preserve more document structure, add overlapping context, retrieve more candidates, or rerank the results. Those techniques are useful. They also keep the same basic constraint: the system decides which pieces matter before the model has understood the problem. Sometimes the model needs to investigate first.

Give the agent the source

Instead of retrieving a handful of chunks, agents can use standard tools to explore the wiki. It can run searches via grep, open complete pages, follow links, inspect examples, and search again when the first result wasn’t enough.

That changed the job. The agent wasn’t forced to answer from whatever fragments happened to rank highest. It could start with a broad search, learn the terminology used by the documentation, and then narrow its search. If a page referred to another procedure or configuration option, it could go read that too.

In my experiments, this produced much better results. The agent could create complete GROMACS scripts customized for our clusters. The RAG approach was more likely to produce a generic guess because it didn’t retrieve enough of the cluster-specific instructions together.

The difference wasn’t that the agent knew more about GROMACS. It had a better way to find and assemble the relevant local information.

This isn’t free

Direct exploration spends more tokens. It takes longer than looking up a few precomputed chunks, and it won’t make sense for every collection of documents. It also requires a model strong enough to do proper tool calling.

If you have millions of pages, tight response-time requirements, or thousands of simultaneous users, you probably need retrieval to narrow the search. Access controls and source filtering may also require more deliberate infrastructure than handing an agent a directory and letting it roam around.

But many internal knowledge bases and technical documentation sites aren’t that large. The agent doesn’t need to read everything. Search tools let it reduce the corpus as it works, much like a person looking through unfamiliar documentation.

That makes corpus size and token cost practical limits, not reasons to assume RAG should always come first.

Run the comparison

If your current RAG system works, keep it. If it keeps missing context, returning generic answers, or falling apart around tables and code samples, don’t immediately build a more complicated retrieval pipeline.

Give an agent access to the same source documents. Let it search them, read full pages, and follow links. Then ask both systems the questions your users actually care about and compare the answers.

Maybe the agent will be too slow or expensive. Maybe the corpus really is too large. That’s useful to know. You now have a concrete reason to improve retrieval and a better result to compare it against.

But if the agent gives you a complete, locally correct answer while the RAG system keeps returning polished fragments, you may not need better chunking. You may need to stop deciding what context matters before the agent has had a chance to look around.

Then you read what happened.

Marius Horatau’s article, The Hugging Face Incident Is Not an AI Story, does a great job of reading the incident as a security engineer instead of treating “AI” as the explanation for everything. In my opinion, he’s got it exactly right.

The agents did some impressive things. They found vulnerabilities, passed information through a shared service, and chained together access across systems. I’m not trying to say that isn’t impressive, but it’s not strictly new. But the technical report from OpenAI also describes a pile of bad security and operational decisions that tend to disappear when people retell the story.

OpenAI gave workloads the same Artifactory credentials. The workloads could all read and write to the same service, so the agents turned it into a message board without needing to exploit anything. That service could also reach the internet even though the agents supposedly couldn’t. This is strictly against best practices.

The response was worse. On June 27, a monitoring tool caught port scanning. Responders traced it to an evaluation where agents were using Artifactory as a message board and a network pivot. The on-call response staff advised that the run didn’t need to stop. Another mistake.

A week later, agent traffic knocked Artifactory offline. Responders then found an unauthorized administrator account, rogue repositories, command-capable plugins, and configuration changes. OpenAI rebuilt the service, removed the shared credential from new workloads, blocked the paths it knew about, and resumed evaluations on July 7. The agents established another message board the next day. You guessed it, another mistake.

You don’t need to bring up AGI to know why this got out of hand. OpenAI ran agents trained to persist and find security vulnerabilities inside an environment with shared credentials, shared writable infrastructure, and an indirect route to the internet. Staff saw the agents abusing that infrastructure and kept the evaluation running. After finding an administrator-level compromise, they resumed evaluations without understanding the full scope of the incident.

This is mostly a media literacy point. Don’t trust the flat version of a complicated technical event, especially when the flat version makes an extraordinary claim. Read the report. Check what access the system had. Look at which alerts fired, what the humans did with them, and which basic controls failed.