Peter Ottsjö with Derya Unutmaz, Episode 1.

Open weights, Anthropic, biosecurity and who controls AI.

PETER OTTSJÖ

In the latest issue of my newsletter BAIO I wrote about an open letter from Microsoft and Meta, Nvidia, OpenAI, Google and others defending open weight models. And after that issue went out, Anthropic CEO Dario Amodei clarified his position. He says that he doesn't want a blanket ban. Models without dangerous capabilities are a public good, while sufficiently capable models, open or closed, should face mandatory safety testing before release. My problem with that is it sounds like openness by permission. Someone must decide which models the public is allowed to have, and the Frontier Labs will inevitably help write those rules.

Derya I think you are on the same page as me on this one. What was your reaction to the letter from Anthropic?

DERYA UNUTMAZ

Yeah, I mean I I think I make my position very clear on X. I've I've been against this from the very beginning. And I also don't think that he clarified anything. He basically has been saying the same things. It's just in a in a kind of a different way, in a more misleading way. And some of it is is disingenuous, very contradictory. you know, Dario Amodei has been saying this back I think in 2023. So as long as his models are a little bit ahead of what's out there, then it's fine.

You know, he he actually, I think he had claimed that even GPT-2 was too dangerous to release, and that's might be one of the reasons he left OpenAI. you know he was involved in development of that, that it wasn't safe. So this narrative about safety, which is obviously an important issue. but they wanted to sort of lock up that within Anthropic. That Anthropic is the only company, Dario Amodei is the only person in the world who cares about safety. No other AI researcher or you know, someone like me who who is all in on AI and I'm in biomedical AI that we don't care about biosecurity, for example.

Or all these big companies, NVIDIA and Google, they don't care about cybersecurity. Only Anthropic does. They know the best. They need to control everything. So this this is the same narrative. It's just you know he he changed this. So I'm I you know we can we can break it down. Especially focusing on the biosecurity part because he's he's always very vague on that. And that is a great way for creating fear and fear mongering because people are like, my god, you know, what if we have another pandemic, if we have these you know, in reality cybersecurity part is much more much more critical.

and they were the ones who were pulled off by the government. Because you know, they were not they were not secure. And they block you know, Fable 5 blocked all of biology, all of medicine, like nothing. I couldn't, if you remember, I couldn't even ask something about cancer, or my own research for that matter. so it's very hypocritical and very in my opinion, very self-serving. but we we we can go in in depth.

PETER OTTSJÖ

Yeah, so by disingenuous, you mean that he's essentially saying that sure, open weights are fine for those worse models that are not as capable. But he doesn't really say that out loud. He say open weights are fine, but for frontier models, they're not. I guess it's really interesting because he has been talking so much about the concentration of power and how he fears concentration of power. But as long as Anthropic has that power, he doesn't seem to have a problem with it.

DERYA UNUTMAZ

No. No absolutely not. And I think we actually lived through that. in my opinion, you know, June was probably the worst month in the age of AI since last several years because we we all felt that you know suddenly wow here is a here's a great model you know Fable 5 is a great model and they can just restrict it whatever the way they find is right I cannot ask about cancer somebody cannot ask ask about their health, or some small company cannot use it for cybersecurity to defend themselves.

And then they you know they they were so out loud about about this is that the government actually went in and said, okay, well we're gonna pull this out. And so for me, it was a kind of a shock. You know, before I was skeptical of Anthropic, but I I didn't see see them as as big of a threat as as I do now. in fact, I had even posted that Dario Amodei as one of the AI heroes, which which I think you know most of the AI engineers who work there are AI heroes. I mean they and and I suspect most of them have good intentions.

But this was a wake up call. I mean it was it was exactly the dystopian future that we don't want to live through. the danger is always about AI authoritarianism, and that's not because of AI, is is whoever controls the AI. the this is the most powerful technology in the history of the world. and it it it cannot be concentrated in the hands of... I mean it doesn't matter whether it's Anthropic, OpenAI or Google. you know, we have to have options in case something like that happens again.

PETER OTTSJÖ

So he often invokes security arguments, cybersecurity, maybe especially his fear of someone getting an uplift from knowledge capabilities these models have in creating bio weapons, for example. How should we handle that, do you feel? How would be a great way to make sure that we're doing that in a safe and secure manner, but also making sure that a lot of people, as many as possible, can have access to the best capabilities?

DERYA UNUTMAZ

So again, I think he's really sort of confounding these issues. you know, sometimes he makes the analogy that AI is like having a nuclear weapon. you know, as if l let's just say that I ask a super intelligent AI model how to build a nuclear bomb, right? that information is available by the way, you know. it's not like I'm gonna start building it in my garage, right? So you know, Iran has been trying to to build these nuclear bombs for for years and yeah, you know how much effort they have to spend and and what they have to go through to to actually make it.

And if there are sort of very bad characters like you know, countries like North Korea who wanna do it, they will do it anyway. So there's there's no way you can you can stop them. So the same thing is is true for biosecurity issues. in fact, when you talk about biosecurity, like what what does that mean, first of all? Because, you know, people just throw this thing out there and say, Okay, well, you know, if there was a model that can create the next pandemic, a virus that will destroy humanity. I mean it's it's a great movie scenario, right?

and so it really touches on the most sort of the sensitive fear of people. Wow, you know, that could happen. We have to we have to limit it. but how could that happen? Let's let's ask that question, right? So you know, we we had a pandemic which was probably partly man made. unintentionally probably, and that did cause a pandemic, right? So you didn't need AI for that. In fact we don't need AI right now to you know make some virus more deadly or spread faster and and so on so forth. But in order to do that, first of all you need incredibly sophisticated laboratory environments.

I mean I work in this field, I I'm I'm pretty well aware of it. and if somebody, a well equipped lab or in China wanted to do that, you know, they they can. Well, why would they do that, first of all, right? So China suffered more from from the pandemic from COVID than than the rest of the world. I mean, or equally as much. and so you can say, well, you know, there there could be some terrorists who want to make a virus to destroy all humanity, you know, because these are an nihilist.

Again, you have to have a very, very sophisticated laboratory environment and you need to Also if you when you're when you're building that you need to protect yourself. Like you need to have a a BSL 3 or BSL 4 level laboratory because the first person you're gonna infect is your yourself, right? If you're building a a virus like that. And so that's that's that that's one part of it. I you know, with the available technology, you can actually take an available virus and you can modify it, you can make it deadly. You don't you really don't need AI.

You just need all these you know capabilities, facilities, and like these gain of function mutations, right, which which which are bent. But you know, in some country, if they wanted to do it, they they can, right? How are you gonna stop them. but you can say okay well what if the the models are so advanced that they can design a completely synthetic virus that doesn't even exist in the nature. And so that that we we don't have the the knowledge set to to create such a virus. Okay. So that that that is a plausible scenario and it could it could potentially happen.

I don't think current models or even the the frontier models are are able to do that right now, but it might. You still have the same kind of a level of requirements for that. It's like saying, okay, well, the models can come up with a better nuclear bomb more effectively. you still need the the wet labs or the hardware or or the infrastructure. but but there there's another point. and the and the point is that if the if the model is so intelligent to be able to come up with a synthetic virus that can cause a pandemic, that means that that model can cure every disease.

I mean, because people don't realize this. To design something like that. Is much more difficult than curing cancer or or or even reversing aging. Because we are not at the at that level of biology where we can just synthetically design something out of nowhere. You can design proteins, you can make proteins more toxic or or you can change the envelope of the virus, make it more infectious. But those are things that that has to be within what's available with in the nature. You're just modifying things and then editing them, making them better to come up with something completely new.

And that's the only thing that would be super dangerous because we we wouldn't know how to how how to deal with it. and so that's that's one thing. That if he had such a model, it's so capable, then it it would have actually cured diseases, not to mention he would have also figured out how to make the vaccine for something that's synthetic, right? Because a virus on its own means nothing. It's not even alive, right? So you need you need the host. And what's host? You need the immune system. You need you need to have the right receptors and and so on and so forth.

The last point I want to make is that you know Dario brings up the Operation Warp Speed during the pandemic. So well, you know, did you see what happened? you know, it took so long to develop vaccines and drugs and you know if if if somebody came up with this, you know, it would be it would be another pandemic, it would be worse. Again, it's so contradictory. It makes absolutely no sense. Even if we had the current AI models back in 2020, beginning, because you know I was involved in this research research and and you know this this is this is what I do.

we would have po possibly stopped the virus from spreading much earlier. Because you can actually simulate all the dynamics of the virus, how it will spread, whether it's going to move from person to person, you can really simulate those things. at least limit its spread and killing of millions of people. because you know what happened? the the reason why the virus spread so quickly and killed so many people, we we were ignorant. We couldn't figure out. You know, is the mortality 1 percent, 3 percent, 0.1 percent? Is it just going to kill the elderly or what what happens to children?

I mean, you know, like Sweden, Sweden did the best decision which everybody criticized. you know, you guys you guys say we're not closing anything, right? and y you t it turns out you were right. and and the rest of the world was like, no, no, we because intuitively you wanna you You think that you can stop it by closing everywhere and so on, which which obviously didn't really have an effect. But if you had the AI, we could have actually modeled this. AI would probably would come up with that Swedish model, you know, by by January or February, rather than a a a year later. And not to mention that we would have actually created the vaccines much, much quicker. The vaccines were created very very quickly. It just took longer to to test them. If you had some sort of a digital twin type of a model where we can simulate and then test the the vaccines quickly, we would have the vaccine by March 2020 not at the end of the year. so again, you know, a model so intelligent, able to create something like that, would would be would actually be better. in creating the defense for for this.

PETER OTTSJÖ

Yeah, I guess the other thing that he is afraid of, or at least he says so, is that some other region, some other country will jump ahead in capabilities and that they would have the power and I guess the thing that scares him is that Let's say that we reach artificial general intelligence or super intelligence and China, which is a country here often names in this context, is the country to have that advantage. Then he says that we might not be ever able to catch up the rest of the world. And so we're in the hands of a Chinese frontier model and everything follows from that. But we haven't seen that.

I'm not defending China here, but what we have seen is Anthropic restricting access to its frontier models and deciding who gets to play.

DERYA UNUTMAZ

That's exactly right. It's the opposite what what he's saying. They're they're all assumptions, right? you you have to ask certain questions when you when you especially vilify a country with 1.5 billion people. Okay? So and this is very dangerous by the way. I mean I'm not defending Chinese government or their authoritarian regime or or or or whatnot. But when you talk about China. China is not you know a few thousand people who are who are ruling that that country. They're 1.5 billion people. And they are people, they're human beings. they are not any less worthy than we are. Okay. there's it's true for the rest of the world.

There are eight billion people, only about 300 million are are in the US. It's true for Europe or whatever. So when you talk like that, you it sounds like Dario Amodei and Anthropic are the only people who care about the safety of the humanity. Right? They are the only ones who can decide. They're the only ones who can decide how the safety should be should be determined, which models should be open, who should get an access. No. China? No. India? No. Europe, no. in fact, a lot of people in US ,no because we know the best. No, you don't know it.

so these these companies who are developing the AI models in China, they are not evil people. Their goal is not to destroy the world, they're not stupid. they care about safety as much as someone working at Anthropic. I'm I'm absolutely sure about that. Why wouldn't they? same true for OpenAI, for Google, for XAI, any of the any of these companies. There might be bad characters in everywhere, but but most people have good intentions. and so that's that's one point. The second more important point is that he talks about we should stop selling chips to China because you know we need to be ahead. Well how is that working out?

They're developing their own chips and actually Jensen from Nvidia he made this point he said this is a foolish thing to do because the what we are doing is pushing them to make their own chips you know most chips are made in Taiwan. Taiwan is is are Chinese as well, by the way.

PETER OTTSJÖ

Yeah.

DERYA UNUTMAZ

You know, it's not like China has any less expertise or know-how or ability to produce these things and they will very soon they will they will have better chips and and they will compete with with everything we have not to mention that even with what they have available They're creating these amazing models that are more effective, more efficient, almost as intelligent as we saw with Kimi K3. You know, that they're approaching the the frontier levels and they don't have the chips, they don't have the resources that we have. So it means that they're very smart. So they will they will get to that point no matter what you do.

You're just actually accelerating that phase.

PETER OTTSJÖ

But now Dario has another thing he wants to ban. Which is distillation, right? It's not enough anymore to ban the export of chips to China. We should also ban the distillation technique…

DERYA UNUTMAZ

Yeah. Yeah.

PETER OTTSJÖ

…which is obviously going to be very difficult.

DERYA UNUTMAZ

Well, so so I think David Sacks made the best point about that. he he said, Well, if they were so worried about distillation, why didn't they prevent it? I mean if their models are so good, you know, they they couldn't figure out how to prevent others from from distilling it, you know? But but even even besides that point What is distillation? Okay. So I'm a professor. I have a lot of knowledge, expertise, and I teach students. I I give them what I know. They are distilling my knowledge and expertise, right? Or you take a textbook, you read it, it's thousands of people contributed to that knowledge set.

You know, they spend enormous amount of resources and efforts to do it. What are you doing? You're distilling that, right? So i it's it's just so ridiculous. they took all available knowledge of humanity, by the way, destroying millions of books, including very rare ones, which which is really I think crime against humanity's heritage. I mean, I'm a rare book collector. I I kind of revere these as as something holy. Because these are these are things from these are artifacts from the past. and they didn't care. They just burned them. so that they can they it's so they they distilled all that knowledge and then they destroyed it as well.

So what kind of a hypocrisy is this? and then the third point is that. I don't think Chinese AI companies need to distill you anymore. Their models are almost as good. you know, they could they could distill each other, right? You know, Kimi K3 can distill GLM and Kwen or Kimi K3 will will produce enough outputs to to distill itself, right? So i it what's the point? so the arguments are so hollow and so superficial and they they're so hypocritical, so baseless that it's it's just ridiculous. but they were somehow able to convince even the US government making it like a crime to to distill.

If it's a crime to distill, then there should be no universities, there should be no schools, there should be no books. because we are distilling constantly from each other, right? it's unbelievable. This is so misanthropic. I I there there's no other company in the world which which is the opposite of what has been named.

PETER OTTSJÖ

Yeah

DERYA UNUTMAZ

Because you want you want what you should say is that I want other companies to distill this so that we can we can advance faster. My I have great models, but you know, Fable 5 is a different taste than GPT 5.6 and then Grok. They should all collaborate, right? we do need to distill from other models so that the these models can become more aligned, can learn from each other, can become better, and then we move we move faster.

PETER OTTSJÖ

Yeah. So another thing that worries Anthropic and Dario Amodei is that, you know, let's say an open weights model is released. It has frontier capabilities. It has been evaluated for all intents and purposes. It seems to be safe, but then, you know, something is discovered through use by the masses. weights are public, we can't recall them, safeguards can be removed and so on and then all hell breaks loose. What's your answer to that scenario that's sometimes being invoked here?

DERYA UNUTMAZ

Yeah, I d I mean of course that th there's always a possibility for that. But but in my opinion it's exactly the opposite scenario that would happen because you do want to find the sort of the the gaps or what could be you know, there are people who hack into these these AI models, right? They jailbreak them. They're actually doing a great service, right? that you know, these these hackers they're finding The loopholes.

I mean that's the whole point of open source, because when you have open source, you have thousands or maybe millions of people trying things that no way Anthropic, OpenAI whatever they they can't do that by themselves, right? It's impossible. In fact, we saw OpenAI had had this model escape and you know ironically try to hack into Hugging Face the open source database and then they had GLM 5.2 which eventually detected it, it couldn't prevent it, but at least it detected it and so on so forth. So so I mean l you can see the irony there. Dario Amodei never ever Talks about the benefit of open source, right?

He completely automatically dismisses, you know, there's no way there's there's anything that open source can do or defend or whatever. but the reality is that open source would make the models much safer because it will find what is, you know, if they made a Fable 5 partly at least open source, right? like you know Elon Musk you know he he he open sources the algorithm for for X, right? and why is he doing that?

So that people can actually improve on it and say, okay, well, you know, maybe you should do it this way, that way, you know, that that everybody can see what the loopholes are, you know, what can be hacked into it. and for transparency, this is extremely extremely important. so I I don't buy that at all. I think it's it's gonna be it's gonna be exactly the the opposite. And why is it that like every other major AI company doesn't agree with him? Why Dario Amodei is the only one who's right or claims to be right? You know, this question should be asked. he might have an opinion.

I respect the opinion. it's fine. You can say this and that. But it it's not that he just has an opinion. He has the power to influence this regulatory capture. This is the problem. Opinions are fine. Everybody can say whatever they want and they they can make their arguments. as long as they don't have the conflict of interest, that's perfectly fine. But he has the conflict of interest of his own company having all the power, making all this money. And and not only that, he has the power to really influence the future of AI and future of humanity, honestly. So this is highly problematic.

PETER OTTSJÖ

Yeah, one thing that has nothing to do with Dario per se, but that came up when I wrote about the letter that the other companies wrote about the importance of open weights is that a few other scientists and thought leaders in the field, Bo Wang, for example, said that, hey, know, this is great, but we shouldn't forget that What biology also needs is data to train the models. What do you feel like we could do there to accelerate data collection and finding the right types of data to have in bio to really accelerate the field?

DERYA UNUTMAZ

I think the data is is the most important thing right now. I mean there it's not maybe the most, it's the compute and the data. So if you don't have enough compute, if you don't have enough data, what's the point of having you know, massive massively big model like Kimi K3? I mean, even to run it locally you need to spend a couple of hundred thousand dollars and you really need need to feed it with with the with a lot of data as you said. so I think you know again ironically or paradoxically having open source actually accelerates having having data.

Look none of the AI companies would be would exist if we didn't have open source data right like I mean it they they all started with Google you know publishing their transformer paper or having the Image net having PyTorch and you know actually Meta did a lot for that. Google you know they released so many open source models and and so on. So how will the researchers actually advance the field if there's nothing open source that they can tinker and they can change and they can produce the data to make it better. And the same thing true for biology, right?

So if I cannot ask a model what would be the top 10 treatments for cancer, I mean this is of course a a kind of a rhetorical question, but you know, or if I cannot upload my own data and come up with some new ideas, then what's the point? Right? yeah, how how am I going to make it better? If I if I'm trying to develop a a vaccine for a virus that may or may not show up, I I'm gonna need to work with these models, right?

and and they can say, well, you know, you can apply for our trusted you know trusted institutions or people and we may or may not... actually I applied just just to see what would happen. I never heard back. I applied to to to Anthropic actually almost two months ago. They didn't even respond like you were rejected or were not a you know and this was this was before I was against the Anthropic so they don't they don't really care. I mean and then the there's there's one more point related to that.

Dario Amodei talks about well, we have to test them, we have to put them on rigorous testing for security, biosecurity, or this and that. Like, what are those tests? Who comes up with that? Who decides what is the threshold? How do you how do you determine a model is safe if you don't even know what could be possible? Like, you know, OpenAI's model, they I'm sure they were trying to sandbox it and make all the all the security limits to it. They they weren't even aware for for a week what was going on.

So If you don't if you don't understand your own model, which nobody does, how how do you going to come up with these you know set of benchmarks that will determine, okay, you know, Kimi K3, yeah, it's approved. It can it can be used. Like w what what is a set of questions that you're going to to ask that to make sure of that? I mean, we don't even know what to ask. Because what

PETER OTTSJÖ

Yeah.

DERYA UNUTMAZ

what could go wrong is things that we don't know. We don't know what we don't know in biology or in cybersecurity. so all of these things are so so vague.

there there's no I think I think you know something like what Demis Hassabis proposed makes a lot more sense that you have a group of people who who are in the in the know who are working with these models and then if you want the biosecurity you need to bring together you know several hundred maybe several thousand scientists who are who are experts in this so they can try to push these models to see if they can break them like what what would be what could happen if you really really push them could they actually develop something that would be dangerous.

but even then it's gonna it's not gonna be very easy because how do you how do you show that that is dangerous? You actually have to do the experiment. You can't just say, well the AI came out with this incredible viral sequence and that will cause pandemic. You don't know that. Right? You you need to test it. How are you gonna test it?

PETER OTTSJÖ

Yeah, I'm sure the debate and discussion and development around this will continue for some time. It's happening as we go in real time here. Thank you so much, Derya for enlightening us today. And yeah, we'll see what happens and we'll continue reporting on it at BAIO of course.

DERYA UNUTMAZ

Sure, sure. I hope we can talk about fun stuff. I really don't wanna talk about this potential dystopic future because I think AI AI is the is the it's the big should be the biggest excitement and hope and optimism for future, not the other way around.