It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".
Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.
It’s not a great loophole per se but it’s a flawless example of what I might call “loophole thinking” which is kind of a transition from individualism to selfishness that has been happening for a long time (maybe there was never much of a difference)?
- Quoted phrase in the opening sentence
- Opening quote: “Only god can judge me”
- Opening sentence uses a quote to challenge or distance a term
- Opening sentence challenges the term “Only god can judge me”
"The average Japanese fears not the judgment of an omniscient God, but the judgment of his or her fellow Japanese."
Here's my most-recent real-world problem: I wanted to plug in my midi controller keyboard and just play a piano, so I can practice my choral music. Very basic.
This turned into many hours over several days trying multiple DAWs, trying to get tehm to install, get them to run at all, get them to recognize my controller, get them to play a VST plugin...changing system settings, completely jacking up my sound, restoring it all again.
It was a nightmare.
So instead I fired up Claude, worked with it for about 3 hours, and now I have an *8-channel* VST player that supports all my hardware, and includes a snappy UI, with outstanding performance and sound. It's amazing!
I literally spent less time creating a custom solution than I did trying to get the off-the-shelf stuff to work. I could make my player OSS too, but the effort to do that is huge, for all the reasons this forum would understand.
Bespoke software avoids all of that and allows me to do what I want, with the environment I have. I don't have all the features of an FL Studio, but I have what I need, and that works for me, beautifully, at low cost and effort. What's not to like?
So maybe we should share OSS "templates" instead of full products, projects that are the barebones of a solution that you can clone, and have your LLM modify to suit your needs, on your platform of choice.
For example, perhaps my barebones VST player is mostly good enough for you, but you also want to play sound fonts? Cool, clone my project, instruct your LLM to add sound font support, and add support for your controller or your output device. Or convert it from Gnome to KDE.
The time and cost of doing this is so little now, why not? I'm getting more out of my computer than ever, and I'm happy about that :)
- Quoted phrase in the opening sentence
- Opening quote: “i don't have the time to deal with this”
Have you considered that the person submitting the PR to fix said problem has themselves experienced it? I find that is most often the case. But there is definitely an increase in the "performance edge cases, or hypotheticals" framing, where I am equally befuddled/too tired.
This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.
There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
1. You can contribute to rural areas in return for a tax credit, effectively allowing you to redirect a portion of your taxes to another town.
2. The most important quirk of the system is that you can choose any towns to contribute to. There is no requirement of previous residence or whatever. In fact, most people contribute to towns with which they have no personal connection.
3. This ended up causing tax competition among local governments. Quite a few towns started advertising expensive "gifts" to contributers ("Give us your taxes and we'll send you some goodies"). It went so far that some towns got banned from the hometwon tax system.
It was introduced with good intension, but resulted in a very inefficient tax system.
Thanks, Andreas Rumpf, for sticking with it for all these years: This seems like a great opportunity for me to get back into nim.
<button class="nav-btn" onclick="navigateToPage('index.html')">Home</button>
...
// Navigation functionality
function navigateToPage(filename) {
window.location.href = filename;
}
What shockingly poor code. Just disgraceful.This is bad for accessibility, bad for compatibility, and serves no purpose at all.
This is the unsolved problem. Neither the platforms nor app developers are aligned on solving it between themselves, much less aligned with user interests. My pet peeves:
- Apps requesting way broader or more general access for reasons that are unexplained, whether due to innocent oversight or actual malice.
- Platforms aggressively segregating access into broad categories (to reduce user fatigue), which compounds the problem of broad requests, and then preventing users from being able to override those.
- Platform giving special bypasses to their "valued partners", and refusing to inform the users or give means to disable this access (or grant to apps of their choice).
Now wrt. platforms, I get the arguments and I'm sympathetic, even though most of them don't apply to majority of customers. The end result is still the same: users don't get to offer informed consent, and have little to no control where it actually matters.
Could you report this: https://github.com/ankitects/anki
----
For historical reasons, the Android app requests "All files access", if not using the Play Store variant. I'm working to fix this.
We don't need it. The replacement API is 300x slower on some phones[1] and is infeasible to use.[0] It should be possible to use AnkiDroid in a "public" folder, but it no longer needs to be the norm. Times have changed
[0] https://news.ycombinator.com/item?id=43987745
[1] just tested: 966ms for a file delete on my Pixel 9 Pro vs 3ms
- Allow the developer to specify the common location(s) that they need access to. Rather than having to have the user select the folder they want to grant access to, have the developer say "this app uses data in these 5 folders, do you want to grant that access" - so much clearer, and useful in cases where an app has to integrate with other apps through the filesystem. - Readonly access. Allow an app to read your files but prevent writes or deletes. - Time limited access. Allow some kind of access and allow the user to auto-expire it rather than having to remember it exists. - Allow for negative access (i.e. allow labels or tags on directories to mark them as sensitive and prevent access by default - browsers could do this for their cookie stores, `~/.ssh` could ship with it as a default, etc.)
Apple could have used this opportunity to think through the question of "why do developers ask for filesystem access, and how can we allow legitimate usage through well scoped access controls", but instead they built a bad API that will (I would guess) not actually allow developers to be more judicious or achieve any of the user-safety benefits.
Apple needs to learn to talk to (and listen to) developers.
1. LLM performance has more or less peaked. Since Fable 5 released in June, every Claude model has been a ~1-2% improvement over the last one (if you go by benchmarks). Fable 5.2 marketing verbiage has shifted to how it's "lower cost per task" which imo marks a regime shift. I would bet money that this time next year, we're not going to have a model that's obviously and significantly better than the best ones we have today.
2. If you hold that this is about as good as LLMs are gonna get, we haven't found ourselves at a point where we can completely remove the software engineer from the seat. A software engineer is still required to create real commercial-quality software. The role hasn't been commodified either - a better software engineer still writes better software.
- Quoted phrase in the opening sentence
- Opening quote: “learn the fundamentals properly”
If you learn the fundamentals properly, then AI will be a power tool for you. If you don’t, then it will replace you.
I don’t mean this in an aggressive “you better get used to it” sense, I realize there’s nuance - but this is the version that fits on a T-shirt.
ISAs are pretty complicated and reasoning about programs in that context is hard which is why we have high level languages.
> This distinction in terminology used to be a colloquially accepted notion, when we heard "CompCert is a formally verified compiler" or "seL4 is a formally verified microkernel" we assumed the authors did not verify vacuous or trivial properties, because the work of verification itself was held to a high standard that highly affected the reputation of the researcher.
CompCert has also had extensive empirical evaluation to vet the theorems. Famously the CSmith paper (https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf) has this rather choice quote:
> As of early 2011, the under-development version of CompCert is the only compiler we have tested for which Csmith cannot find wrong-code errors. This is not for lack of trying: we have devoted about six CPU-years to the task. The apparent unbreakability of CompCert supports a strong argument that developing compiler optimizations within a proof framework, where safety checks are explicit and machine-checked, has tangible benefits for compiler users.
At my work we've extended this to CPU millennia. I hasten to add this of course doesn't rule out the possibility of poorly formed assumptions in the semantics etc (it's empirical after all) but it helps to build evidence that we have the theorems and semantic design right. More generally differential testing of this sort offers a possible way forward to build evidence in favor of theorem statements beyond expert examination.
> Every discussion of formal verification must come with a list of caveats, a precise understanding of what is trusted, what is proven and what is not.
Anyone that seriously participates in this work practices this. The whole community. More importantly this is precisely the point of writing down a theorem so that we can closely examine the assumptions. To abuse the CompCert example yet further it's now entirely possible to draw the envelope of assumptions, which are many for a non-toy compiler: https://arxiv.org/html/2201.10280v2.
Last note, I like the kiki bouba diagram if only because it evinces the problematic "round cow" assumption and how that doesn't pan out for real software.
“Ready? 10 seconds to power on server alpha-romero-one-o-four!”
“Load—at five percent, check! All systems stable!”
If the goal truly is social progress, perhaps we should spend a little less energy hunting heretics and a little more effort showing empathy. It doesn't matter that the empathy isn't reciprocated. There's value in being the more likeable side to anyone watching from the sidelines.
It's not lost on me that our side often responds to hateful language with our own forms of hatred. Both sides are complicit in allowing these differences to calcify into the worst kind of political tribalism, but I think our side fails every time it turns a teachable moment into a "hateable moment".
Hate cannot drive out hate; only love can do that.
- Quoted phrase in the opening sentence
- Opening quote: “problematic”
- Quoted phrase in the opening sentence
- Opening quote: “anti-profit.”
- Quoted phrase in the opening sentence
- Opening quote: “a little copying is better than a little dependency”
1. Search for "temporary email", there are zillions of these. Some services detect that they are special temporary emails, many don't. They have tons of domains, always creating new ones, then you can get a random email on that domain. Many of them even let you log back in later if you save off the password.
2. Use duck duck go - Lots of folks must be using ddg to create a new email that hides their underlying email, even my non technical spouse uses it. Most sites accept this today, even reddit.
3. catch all domains. If you have your own domain, you can set it up so <anything>@your_domain.com will go to you. With the latest "advance" from the modern world's anonymization destroying stuff, you have to figure out how to setup dkim and dmarc etc.
I think we'll all be polluting our account/pw software with thousands of these. A huge hunk of junk.
- Quoted phrase in the opening sentence
- Opening quote: “gotcha”
Yeah, if he wants to? If he finds value in that? Why wouldn't he?
It's his voluntary, willful choice. Seems like he was fine doing it for decades.
And sounds like after 20+ years, the site's usefulness had come to an end, and maybe the time investment wasn't worth it either. This did happen within a shadow of that big php vuln.
There's more to life and building things than monetary success. If you struggle with this concept, I suppose there's not much more of a conversation to be had here.
I needed it today when I stumbled on yet another "create a free account to read this article" page and noticed they had closed. Checking the Wikipedia talk page and Reddit it seemed they had closed today.
It doesn't cost them anything but a "whoops" post when they fuck up, so the incentive is misaligned (as you noted under repercussions).
The cost is externalised onto everyone else not them so it's not going to change.
- Quoted phrase in the opening sentence
- Opening quote: “tags”
I hope folks are re-evaluating that now.
- Quoted phrase in the opening sentence
- Opening quote: “tags”
I hope folks are re-evaluating that now.
Woah now, let's not get too controversial.
- Quoted phrase in the opening sentence
- Opening quote: “knowing HTML”
Sometimes you do want to represent the concept of “this whole datatype OR nothing”. How do you suggest this is modelled? I think optionality is pretty fundamental. What’s wrong about classical null is that it ascribes optionality to all types regardless of whether they need it.
Being forced to represent optionality with a “special” default is terrible design IMO - see the recent SSH RCE on Musl, largely due to the poor choice of indicating “crypt failed”. Default values can be quite dangerous. Sometimes the domain you are modelling supports such a value, but having “default value” be the default (and not a choice) is a similar evil to default null.
We've allowed them to grow this way by preferring convenience.
Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
If only.
The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
- Quoted phrase in the opening sentence
- Opening quote: “Zero Data Retention”
- Opening sentence uses a quote to challenge or distance a term
- Opening sentence challenges the term “Zero Data Retention”
Also, I don't think it's that "AI code is inferior" but rather devs got sloppy and no longer review code. Not to say that everyone's lazy but me---a lot of systemic/environmental factors could be the reason.
- Quoted phrase in the opening sentence
- Opening quote: “I have a gun and I’ll be good scenario”
- Quoted phrase in the opening sentence
- Opening quote: “straw that broke the camel's back”
Chernobyl was caused by a guy flipping a switch, Challenger blew up because of a faulty O-ring, etc.
When banks started moving to computerized systems, maybe a bank would be unable to process something... but not multiple banks. I can't recall any "outages" that affected banks nationwide for any period of time. Walk into your local bank and find out they can't do something because their big system at the mothership is down? Perhaps.
Consolidation likely plays a part in this as well. I can't speak to the FAA/ATC systems 50 years ago, I have no recollection of what they were like then, but the only nationwide system I can think of around that era was Ma Bell. And I don't honestly recall any nationwide outages back then. Regional, sure. You might have a problem with part of the country getting calls if there was some kind of emergency... (e.g., everybody trying to call their grandmas in Kansas after reports of a tornado) but I don't remember there being any nation-wide problems like that.
We had to count on actual snow for snowdays. :-)
- Quoted phrase in the opening sentence
- Opening quote: “entire monetary system”
Things like food, water, basic sanitary items, would be bartered for.
- Quoted phrase in the opening sentence
- Opening quote: “according to the website downdetector.com”
Ugh. DownDetector's methodology is… not great.
Steam isn't down: https://downdetector.com/status/steam/
Amex isn't down: https://downdetector.com/status/americanexpress/
Visa isn't down: https://downdetector.com/status/visa/
AWS isn't down: https://downdetector.com/status/aws-amazon-web-services/
But each has a huge corresponding spike there.
I think this is the completely wrong takeaway.
Ober the last ~15 years I have probably used around ~10 web frameworks/libraries
_not a single one was even close to being complete or mostly correct, even just for standard HTTP ~1.1 handling_!!! (The routing, path, header handling. Not any of the complex stuff in HTTP2, Quick etc.)
the problem isn't using dependencies, that is the solution (not reinventing the wheel again).
And preferring reusing solutions which are more complete, practical testet and have more resources to maintain/improve basic functionality is (in most situations) not "anxious" development, but (should be) common sense.
The amount of hidden complexity in even somewhat basic stuff like HTTP is enourmous, rewriting everything all the time is wasting billions across the industry all the time and (in general) increases the chance for bugs.
The problem isn't "using dependencies" or small dependencies, but:
- availability (or lack) of high quality well maintained dependencies
- "no one" wanting to pay for software dependencies
- misguided tendencies for 'clever' & overengineered solution in one extreme (e.g. the stuff which lead to log4shell)
- misguided obsession with deceptive simplicity in artificial examples which lead to 1001 incomplete solutions because of the obsessiv simplicity doesn't stop at cutting out over-engineering but also cuts out fundamental parts
- lack of widely adopted, effective, tooling & infrastructure to trust dependencies.
- lack of tooling to more efficient reuse and compose working parts, the way we do programming seems just very fundamental broken.
- also a lot of legacy tech we are stuck with making many things unnecessary complicated
Also as a side note, number of packages != number of entities to trust. Weather you use 10 or 1 package from the same source, maintained and published in the same way often makes little to no difference for supply chain security.
The PHP2Shell bug existed more than 15 years ago, spanning releases as early as 5.x. There was no AI coding back then. So it's very hard to blame AI. There was a coding mistake, and AI found it in 2026. Probably a good thing before a malicious party does the same.
OSS definitely flourished way before 10-15 years ago. I'm not sure what the author suggests as "knowing the details". Should we all rebuild our own web scripting language to know "all the details": e.g. PHP? Probably not practical. A line has to be drawn on where your knowledge and expertise stop. THe line can be vague and consistently changing based on needs. But there will always be a line. We will always stand on shoulders of of our predecessors one way or another
I understand the author's trying to say the overall situation is unstable and trending negative but I am not sure the writing's logical flow is holding up very well.
A better way to put it is we've written a lot of code, and a lot of it had bugs hiding in plain sight. AI made the whole bill come due at once. And will this AI code explosion create another bill we won't know about until 20 years later? Who knows.
1. Consolidating internet-dependent projects to a few winning cloud vendors and a dozen winning SaaS providers, made all of those project have coincident failure points. Which is great for corporate accountability ("it's Amazon's issue, not ours!") but bad for end users ("shit, EVERYTHING is down today")
2. The movement to migrate fat local-first projects towards thin internet-dependent or server-hosted ones made almost all software share those same coincident failure paths. This was good for entrenching a subscription business model and brought some end user conveniences, but meant everything being down meant everything being down.
3. Anxious and lazy dependency-first development culture ("you could never write that as well as some github rando") meant projects now had an order of magnitude more contributors but only reviewed and owned code for a surface level, making all these projects extremely vulnerable to supply chain failures and subject to constant maintenance churn.
4. Employment and team tenure continually fell so that most projects no longer have experienced owners and instead just temporary custodians.
You can find the seeds of many those problems in the 1990's and earlier and you can even argue that some of them were inevitable (and even worthwhile) destinations for the industry as it evolved without an engineering guild to slow it down and insist on rigor or evidenced safety.
But anybody who's been around for a few decades can attest to how these risky patterns rapidly grew into norms over the past 20 years and shifted the industry from one where any given software had some idiosyncratic bugs that users could learn to work around to one where all software had lots of bugs, many of them correlated across apps, whete failures would wash through and shut down (or make vulnerable) whole industries for hours or days, with effecrively nobody in particular to hold to account for any of it.
Software has never been in a pristine "golden age", but the last 20 years built almost everything on one giant, shared house of cards that's now teetering on the brink of collapse. Better or wose, it's indisputably different than it was before.
As a result I get:
1) The enmity of my lead dev who prefers to do things more fast and loose.
2) Not promoted, due to the first reason and due to the fact that I don't have "impactful" metrics.
I'm not saying I'm perfect, there are definitely things I could have done better. But the fact still remains that the industry grossly undervalues maintenance and optimization that doesn't lead to easily measurable numbers that impact the bottom line.
That in turn is because it's hard to tell if SWEs are actually paying down critical debt or just wasting time. But that's the core problem, not that we're not good at it.
The good news is I can get AI agents to do this kind of work so I can now chase after metrics.
AI is great at following patterns. AI is great at writing tests. AI is great at doing work that humans would consider drudgery and then applying "enough" judgement to it.
The problem is that most software teams and most engineers are very, very bad at the thing that AI needs to succeed: technical writing. Documenting the patterns, the rules, the heuristics, the SOP, the judgement; developers could barely be bothered to write comments before (which in the AI age are free, infrastructure-less memory that is always "line-of-sight" for the agent and can be used to enrich with keywords).
It's not that AI is the problem, it's that we humans are good at resolving these constraints in the real world at a human pace. AI makes that pace impossible for a human to keep up as a trailing activity. Rather teams need to invest more in technical writing and producing the corpus that AI relies on to learn the operational constraints and rules.
Everyone loves to harp on about how terrible the Registry is, but that’s missing the forest for a tree.
The registry + GPO engine is a strongly typed and layered settings engine with:
Machine, Machine Policy, User, and User Policy layered on top of each other, consistently, through pretty much every native app!
More importantly, policy would apply or unapply dynamically and cleanly.
You could move a laptop to a different office and soon as it connected the wifi its set of configured printers would change, its outbound proxy could be reconfigured, etc.
More importantly: Microsoft engineers of the era put in the work of writing up thousands of settings in template files that shipped with Windows, with simple checkbox UI controls and paragraphs of descriptions in every language!
If you think your terrible “settings automation” tool is a viable alternative just ask yourself this question: does it “put the work in” or does its user have to?
Machine accounts have limited rights compared to regular users. AD provides fine grained distributed policy and cert managment. After decades of emperical refinement, AD is often superior to what the cloud offers including Entra. In environments with many PCs that require centrally managed policy, AD is still relavant and probably required.
I think it is also true that there are and always will be many companies that simply cannot expose these PCs to the cloud either because they're working with data that cannot be in the cloud like legal files, finanical documents, schematics or they cannot risk being tethered to the Internet. You know that little micro PC attached to the cart in every exam room in a doctors office? There is a 99% chance it is joined to AD.
> Human approval doesn’t scale to developers creating and deprecating testing
> databases, Kubernetes clusters, and CI runners at will. Contemporary AD
> implementations deal with this by deferring authority. An admin token (sometimes
> hardcoded…) automatically joins new identities. [...]
> Even if you built all of the complex automations to reflect every testing database
> and Kubernetes pod in AD with automated admin approval, it’s security theater. The
> unified directory would exist on paper, but lack authority. [...]
> Modern cloud infrastructure automatically asserts machine identity: [...] In
> Azure, you can configure a machine identity in Entra ID and let Azure’s Instance
> Metadata Service (IMDS) automatically issue OAuth tokens when machines request
> them.
Okay, so configuring machine identities manually in AD doesn't "scale" to the container age, fine. But configuring machine identities in EntraID does? How is auto-issueing identities in AD "security theater", but auto-issueing identities in EntraID the way to go?I believe what you're trying to say there, with "drop in replacement", is that jolt emulates common Java interop forms in Clojure JVM such that code written for Clojure JVM doesn't need to be ported to run on jolt. This a compelling feature which other dialects are tackling as well.
What I would mention as a key difference between jank and jolt, aside from the C++ story and native project management story, is that jolt is very much AI coded and jank is not. jolt doesn't disclose this on its website or Github readme, but it's something which clearly matters to a lot of people.
One thing to note is that the framing "Just 400 US Data Centers" makes it seem like 400 out of the gazillion data centers use this much electricity, but the original research seems to be that they found exactly 403 "Hyperscale Data Centers" and evaluated them.
The framing is technically correct (since there exist non-hyperscale data centers) but is clearly intended to be deceptive.
[1] https://hsph.harvard.edu/news/hyperscale-data-centers-energy...
God I miss being able to walk into a store to find a quality item that I need. Nowadays it's really only Microcenter where you can decide "I need this thing" and come home with that thing.
At some point, we lost the plot on online shopping. Sometimes physically going to the store to hold an item in your hand and bring it home within the hour is far more convenient than comparing 4-8 items online and waiting several days for it to arrive.
We really lost a lot when almost all shopping went online.
- Quoted phrase in the opening sentence
- Opening quote: “Black Friday”
Black Friday predates Amazon by decades, by the way.
Most online retailers (Amazon included) still give free shipping above a certain threshold ($35 per order or whatever) so for things I need, I wait until I reach that threshold to place the order, and often purchase at other retailers or directly from the brand.
Also, everyone does "Black Friday" sales nowadays, in many cases matching the same window as Amazon.
- Quoted phrase in the opening sentence
- Opening quote: “sale”
20% Off! 8€ instead of 10€ (lowest price in the last 30 days: 5€)
and you know it's a junk sale.
Walk into any store (JC Penny etc) half the items are on sale. Even groceries - go to Safeway, there is a newspaper style coupons - multiple pages. This whole thing is just bizarre and tiring.
- Quoted phrase in the opening sentence
- Opening quote: “sales”
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.