A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or
B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.
If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
Do you really think we can really get every country to truly pace the frontier? Pretty sure China won't give a f until they catch up Anthropic and OpenAI. It is an arm race. We had nukes for like 70 years and still haven't figured out how to make every single country follow those nuclear treaties, with an increasingly non-interventionist US I don't think we can get every single country to the table and agree to a pause. Will US accept their frontier being caught up by Chinese Labs? I don't think so.
We don't need every country to pace the frontier, just a certain few. And yes, I believe it's possible and the prior art is nuclear non-proliferation. Non-proliferation wasn't perfect, of course, but it was good enough (so far) to pull back from the brink of extinction.
Of course it is B). If you can use LLMs to find bugs and vulns in software you don't own, you certainly can use them to find bugs in software you _do_ own. They are amazing at that job.
Also, models like GLM 5.3 have zero guardrails in that regard ("find vulns in (...)" prompts just work)
> Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality
Smashing the gas pedal is not "taking appropriate measures". Did you read the article? Needing more time for more secure operations is literally one of the arguments for pacing.
One shot extinction? Seems unlikely. "A lot of damage" - more likely.
The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous.
After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place.
Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario.
Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things.
At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.
This is a different situation. A doctor is a domain expert, but you are in possession of vastly more empirical data about your specific body. Getting to a good result is going to be easier with collaboration, rather than one of you confidently telling the other what the problem/solution is.
This is a really good point. My wife and I produce a history podcast where, right now, a lot of it draws on primary sources such as letters, diaries, etc. from the late 1800's to early 1900's. The ones in German she translates into English, but it feels "wrong" to change someone's words that are already in English.
The net result is that it's easier (for me) to understand a letter written by someone in a translated foreign language than one written by someone in English from the same period.
It's hard to pick a favorite - everyone brings their own perspective to their time. One of my early favorites was 'A Jew Who Stood Guard for Hitler'[1], the account of a kid who joined the Nazis for a while - till they found out he was Jewish and kicked him out. Ah, the naivety of youth.
Yeah? And here I've been a happy Transkribus customer for some time now. If there are better models or interfaces out there for analyzing historical handwriting, I'll definitely take a look.
In the movie Sneakers, a whole scene is taken up sending some guy on a date with Mary McDonnell so she could record clips of his voice. Today she'd just need a phone call or his Instagram. It's getting harder to keep up with who _people_ are online, much less organizations and domain names.
I don't understand what your point is. Do you disagree with any of the concrete suggestions in the blog post about what should have been done differently, or do you think they're hard to follow?
A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or
B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality.
If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
reply