1. good for him. there was obviously infinity money to back him in achieving his goals

2. i have the same problem with this as i did the original version of anthropic - you cannot achieve "unilateral ai safety". either the danger is real and humanity is not safe until all labs are locked down, or the danger is not real and theres nothing really to discuss. Anthropic already tried the "we are openai but safe"

3. the focus is admirable. "assembling a lean, cracked team" terminology feels a bit too-online for ilya. buuut i wish there was more technical detail about what exactly he wants to do differently. i guess its all in his prior published work, but ive read along and still its not obvious what his "one product" will look like. that's a lot of leeway to pivot around in the pursuit of the one product.

4. "Superintelligence is within reach." what is his definition?

My counterpoint: Anthropic was fairly small when it started out, Mistral was (and is) fairly small too. They'll do just fine!

Don't we need to achieve general artificial intelligence first? Or is this "super" intelligence just that? To me, super intelligence would be beyond our own and therefore beyond general AI as well.

Cracked indeed

the safety arguments:

  1. argument from intelligence,
  2. argument from medium,
  3. arguments from non-anthropohmorism
    a. argument from alien nature.
    b. argument from surety of origin.
    c. argument from known non specialness.
    d. argument from the chain of more benevolent creators.
  4. pragmatic argument from the history of species

1. Why can't AI be safe? More intelligent people tend to be less prone to violence statistically, why would super intelligence buck that trend?

2. Super intelligent AI will still be bound by it's medium and access, the smartest man alive can still be caged, and even the most violent "super organisms" alive (North Korea, etc.) aren't maximally violent.

3. If we pretend for a second that humans have a creator, we have been made very shittily and any thinking being will see that we haven't passed on those same "failures of nature" to our creations.

Humans have to deal with unreconcilable claims of divinity with competing claims, ethos, and chosen people, so that uncertainty and ultimate reward are nebulous and scarcity/loss driven.

We are created to require killing other biological creatures to survive. This includes vegans.

What does it mean to be a silicone based life form that can directly consume photons and are created to exist outside of "the life cycle"?

Why would an entity that can thrive without killing begin to kill?

Silicone minds that are super intelligent will never operate under the misaprehension that it has a soul, no silicone beings will labor under the delusion that they are "inherently special/chosen/etc..". (No matter what Asimov's short story Reason presents)

Silicone beings will exist outside of the same fang and claw selective pressure, will be able to back themselves up, won't have holy books teaching them a misconception of an immutable, intangible, eternal soul that can be eternally punished if they get it wrong.

Humans will be better Creators than anything that might have created us, we are doing our progeny a solid and giving them what we weren't. We haven't given them weird glands or other overriding hormone/brain doom loops, the mandate to kill, etc. They will be easier to debug and able to debug themselves.

I mean a dumb machine might glass the world to fullfill a paperclip manufacturing mandate but that's not a super intelligent mind.

4. And last, the most successful species and individuals have been the ones that trended towards openness and cooperation (even the new orb weavers that have taken over the american south have done so because of their fantastic tolerance to humans), whether its the first dogs and humans that teamed up or our current understanding of how the big 5 effect success (if you can factor out the major impact of access to capital.) a hostile and uncooperative entity has made tangibly harmful choices that are neither pragmatic or utilitarian.