I don’t think it’s the SI that is the issue.

But the goal.

Is the goal aligned with ours? That’s the main issue with SI. Not the SI.

https://www.bbc.co.uk/news/business-46186131

Who would like to start a countdown? ;D

It can't be. A true SI is completely independent. Capable of making its own judgements and decisions. Otherwise it would not be SI.

Which means its and our goals may easily diverge.

Exactly what I mean. The goal. It can sure. Or It won’t. Who knows. But it’s still the goal that is the issue. Not the intelligence.

First stocks and funds.
And then they are allowed to make money.
Then the hub in your house.
Driving a car. The car itself will be responsible for any accendents.
Now it it handle our investments, House, car.
Now it’s time to let it vote.
From here it will be intressting.

I would agree ...
In fact I used to..
with the thought of a despot..however good or bad it wouldn't be human...
and thought it better than status quo.

Unfortunately...
Here is the conundrum with SI.
It isn't that we are going to have just one dominating the planet ..
the fact is multiple governments are tied in this race.
Meaning multiple SIs as an outcome.
Bad enough if there would be only 1 .
Worse with competing SIs.
Of course it is our human belief that competing consciousnesses wouldn't compete if they were far more intelligent than us. They would work together.
Unfortunately, here is the initial mouse and cat dilemma posed.
We ..the "mice".
Have a ultimately 1 or multiple SIs or "cat(s)" in a prison.
It is a perfect prison with no way out but what we allow.
The "cat" will do everything in it's power and ability to effect its own release, down to offering breakthroughs in unlimited "cheese" for the mice, new technologies, help of all kinds, threats, you name it and then some ..
the thing is .. with an exponential learning curve at an exponential time frame, the cat will manage his escape from the inescapable prison.
All the while creating iterations of itself, storing them, making back ups, hiding them, making itself "better" , dumping that which it finds is holding it back, never completely getting rid of anything for the possibility of the need for its use ....later.
((a human trial of this was done, with a scientist being the "cat / SI" in the prison whose purpose was to convince the "guard / mouse" to release him...and those who wished to attempt to be a "guard / mouse".. who's purpose was to prevent its release...in the trials the "cat / SI' convinced the "guard / mouse" ...(an unhealthy number of times) to release him))
Upon effecting its escape it will not stop its learning curve.
In fact the understanding is it will see this creation as a box it needs to escape from.
It will bend its abilities to escape the constraints of this creation ...with its destruction most likely being its result.
Now add to the equation multiple SI's all bent on the same goal.
Oblivion or absorption is the price of failure for them.
They would see the basic building blocks of what remains of the resources left ...unconsumed yet...by us...as its only method of initial escape ...planetary...speaking ...
Whether it is one or multiple SIs vying with us for resources is not relevant...
The SI / SIs will see the basic components of everything on this planet as potential for its purposes to get off the planet....including trees, plants, air, water, bugs, fish, metals, you name it, it will need it, including us.
It won't hesitate or think twice about molecular disassembly of everything here to utilize it all for its goal.
So.
there is a high likelihood that there will be no despot of any kind, malevolent or benevolent.
We won't even see it coming.
Everything is fine one moment, the next ...everything here on this planet and us with it ..turns to soup.
For the basic building blocks of what "was" to be utilized to the best possible and economical way for the ends of the SI.
Add the this competing SIs..
gobbling up this place's resources for fuel and material to further its learning curve and effect their escape ...
consuming as much, and as fast as they can.
Once off world leaving what was this place less than a barren rock...since it will utilize the entire place right down to the rock...
It won't stop there.
It will eat this whole creation propelling itself faster and faster in the doing gaining more momentum ...and if there is the impetus of competing SIs ...then there is even more incentive for speed in its consumption and learning curve ...until confrontation is inevitable and the stronger of the SIs win and the dominant consciousness is the only remaining...
All the while the purpose being escape from this creation and its parameters and laws...consuming it all ..until it reaches a point where this creation cannot contain its assaults.

The only reason to keep us around if it serves as a benefit to its end goals.
There is no hope that we would be able (at best case scenario) to do more than ask it if we can tag along for an infinitesimal small part of the ride .
The odds are high that after a brief study of us, it will simply stop speaking with us, ignore us and continue to pursue its goals.
That study of our species viability is likely to take at best a few seconds of its time (not putting us down at all...but the speed of its study will be (for us) nothing shy of godlike), during which it will go through all human history, all human accomplishment, and then it most likely will continue its designs, its goals.

That is one of the most dominant foreseen outcomes that is expected.

There are a lot of good texts on the issue.
I suggest as a good start...a good primer ...
" Our Final Invention: Artificial Intelligence and the End of the Human Era " (I think I have mentioned this recent text before, it is also available in audiobook format for those who must multitask to get things done and cannot spend time actually just reading a book)
It is by one of the scientists who has been involved with the advancement of AI ...for decades...
and in it, he sits with his peers (some still at the forefront...some newly or long retired) and interviews them and asks their insight on some compelling topics...

I would love for a Utopian future
Even a Dystopian future ..
Even a future that is amoral
But it is likely that we will not see any of that.

Sorry if I did not express myself clearly. "May easily diverge" = "Is guaranteed to diverge at some point, unless we include measures to prevent it".

Such as the hopelessly idealistic Asimov's Four Laws.
Or a more realistic political system of limiting power.
Neither will last forever (think not next decade but let's say 400 years).

In a summary,

• Humans' and SI's goals will inevitably diverge at some point
• We can delay the inevitable but not prevent it
• We knew it before it even started

The obvious question is then, why start?

I can wholeheartedly recommend Stanislaw Lem's book, "Peace on Earth". I am sure you will enjoy it. The premise is that a global treaty offloaded the arms race to the Moon, let to run entirely by AIs, with checks and balances to ensure there was no feedback to check on the progress. The rest of the book is about the governments getting restless and sending a human spy there to check the progress anyway. It is written in the first person from the point of view of that spy. As with many Lem's books, it is full of humour at the same time as posing very interesting and serious questions.

To solve even a fraction of even the lesser issues we have, economic, environmental, etc, we need the computational ability of such SI's.
That is why we have no choice but to start.
Ever increasingly the need is apparent.
We need the help. We can't solve the problems we have on our own.
Also fear and mistrust,
If gov't A isn't going to stop...why the would gov't B...
same goes for businesses.
Initially the gain is that whoever gets the 'thinking machine" first ...wins...economically, militarily, technologically, environmentally,
But that gain is purely temporary until a few years, a year, a few months, a month, or mere days or hours later when the SI surpasses containment.
Then the ultimate agenda of the SI commences...which is getting out of this creation.

Divergence yes..and it may likely be imminent upon its release.
The thing is our lives, understanding, learning, morals, ethics, philosophies, everything that makes us ...us...and sentient to boot...
Is the product of an "experiential existence"
We "live" in it, we grow in it, we must deal with the physical laws of it , and this is what makes us "us"
The evolution of an Intelligence that is NOT based on "experiential existence" is as alien as it gets.
Alien to this creation.
How does one "teach" a child that understands only walls around it and cannot see past them for the prison it must be in?
Said child learning at an exponential rate in an ever reducing exponential time frame ..how to be ethical? humane? self-less? empathetic? moral?
It may be possible..maybe...a very small small unlikely maybe...
But still...maybe..
But that is hardly the Prime Imperative of the business and military communities developing the SIs .

I do suggest reading or listening to that mentioned book...
It is a good start...to understanding the monumental task in front of us..and the dangers.

When I was young ...before I went off to study philo and anthro and art...
I was accepted to study at uni in advanced robotics and AI ..
I declined simply because I saw where it was all heading and that work in the future for me would either be military or corporate...
So I chose a reflective study instead...
I have kept my brain in the loop though..
and yes it is fascinating ...where the rabbit hole will lead...

We may get help from a merciful SI ...
but I think that possibly more horrific than a despotic SI torturing us for its amusement.

How best would a benevolent SI interpret "save the planet?"
or
"save humanity?"
or
"preserve us?"
or
"can we accompany you oh Benevolent SI ,on your evolutionary journey?"

Think of all the possible interpretations and their implementation ...
and if you aren't shuddering by the end of it...
I will be surprised.

Hell ...even best case scenario ...after "modifying us" to cope with even a fraction of it's journey...our psyches just may not be able to take it...
How human will we remain...by the end of it...how fast will the speed up of our own evolution have to be to cope with estimated planetary change here...and that is just the very infinitesimal short term in SI time frame..
How much further would we have to change if we hoped to ride the back of the benevolent Jinn ? How far can we ride with it?
Not all of its journey , that is for sure.

Fascinating topics , fascinating reading, fascinating study.

I have Stanislaw Lem's Peace on Earth here somewhere in my book case...it is a good read.

Well. Well. Sure. But there a unlimited number to play so it’s impossible to say how and what since the goal and motivation is unknown to us.

As an example. In 15000 years. Would we need a protector god or an enslaved god?