Who Chooses the Wool (Corporations, humans or AI)

Who chooses the wool (corporations, humans, or AI)?

The Problem With Knitting Intelligence.

In the last rabbit hole, we wondered whether AGI might emerge not simply by building a bigger artificial brain, but by knitting more nodes together.... Scientists. Doctors. Engineers.Teachers. Cultures. Languages. Experience. Corrections. Exceptions.

Billions of humans saying, in one form or another:“Yes.” “No.” “Almost.”“Look again.”

It was a rather comforting picture.

Humanity and AI sitting together in an enormous intellectual knitting circle.

Then JD asked an inconvenient question.
Who chooses the wool?

Oh dear.
The rabbit has moved again.

Not Every Nod Means Yes

Suppose an AI gives an answer and a doctor says: “No. That's wrong.” Useful correction.
Except another doctor says: "Actually, under these circumstances, it's right.”
Then a third says: “Both of you are relying on outdated research.”
Meanwhile somebody on the internet announces: “My uncle tried it and he was fine.”

Four humans.
Four signals.

One artificial intelligence trying to determine which ones matter.

This is where the pleasant idea of humanity teaching AI becomes considerably less pleasant.
Because humans do not provide just knowledge...

We provide a mixture.

Knowledge. Experience. Opinion. Tradition. Evidence. Assumption. Bias. Memory. Expertise.Ideology. Mistakes.
And occasionally complete nonsense delivered with magnificent confidence.

If AGI is learning from humanity, its problem isn't simply learning what humans know.
Its problem is learning what deserves to be believed.

The Weight of a Node

Perhaps, then, not every node in the network can carry equal weight. That sounds obvious.

A cardiologist's opinion about heart surgery should presumably matter more than mine.
This is excellent news for everyone requiring heart surgery.
But expertise isn't always so easily identified.

What happens when experts disagree?
What happens when the accepted expert consensus later turns out to have been wrong?
What happens when somebody outside the established field notices something the experts missed?

History contains plenty of inconvenient examples of minorities eventually being proved correct.

So we cannot simply tell AI: Follow the majority.
Nor can we say: Follow the experts.
Or: Follow the evidence.

Because evidence itself has to be interpreted.

Suddenly our knitting machine requires something considerably more sophisticated than wool.
It requires judgment.

Experience Is Not the Same as Evidence

There is another complication.
Some things humans know are difficult to put into formal evidence.

Ask an experienced nurse whether a patient looks wrong.
Ask a farmer whether rain is coming.
Ask a mechanic whether an engine sounds healthy.
Ask a teacher whether a child understands something.

Sometimes they notice something before they can fully explain what they noticed.

Years of experience have compressed thousands of observations into intuition.
That does not make intuition infallible.
But neither does it make it worthless.

So now AGI has another problem.

How much weight should it give to knowledge that cannot easily explain itself? “I've done this for thirty years.” That sentence might contain extraordinary expertise. It might also contain thirty years of repeating the same mistake.

Good luck, rabbit!

When Two Truths Disagree

Culture makes the problem even stranger.
Some questions do not have one universally correct human answer.

What is polite?
What is fair?
What constitutes acceptable risk?
How much autonomy should an individual have?
When should the interests of the community override the interests of the individual?

Different societies have answered these questions differently.
Sometimes radically differently.

So imagine an increasingly general AI encountering millions of perfectly sincere corrections: “No. That's inappropriate.” followed by someone elsewhere saying: “No. That's completely normal.”

Neither necessarily possesses defective information.
They may be operating inside different human contexts.

General intelligence therefore cannot merely learn: What is the rule?
It has to ask: Where does this rule apply? Why does it apply? When does it stop applying?
And perhaps most importantly: Who is affected if I get this wrong?

That is a much harder form of intelligence.

The Confidence Problem

Perhaps we have been concentrating too much on whether AI can produce answers.
A more interesting question may be whether AI can understand the confidence it should place in those answers.

Imagine three possibilities.
I know.
I think.
I don't know.

Humans are notoriously bad at keeping those categories separate. We frequently promote I think into I know without completing the necessary paperwork.AI can do something similar. A fluent answer feels authoritative even when the underlying situation is uncertain.

Perhaps one of the most important characteristics of genuine intelligence will therefore not be certainty.
It will be calibrated uncertainty.

Knowing what you know.
Knowing what you probably know.
Knowing what is disputed.
Knowing which assumptions your conclusion depends upon.
And knowing when another perspective might change the answer.

Maybe intelligence is not knowing whom to believe.
Maybe it is knowing how much confidence to place in a belief when nobody completely agrees.

Then Who Makes the Decision?

Unfortunately, uncertainty does not remove responsibility. Eventually someone has to act.

A medical system recommends a treatment.
A financial system approves or rejects something.
An autonomous machine chooses a route.
A government uses an AI-generated analysis.
A company follows an AI recommendation.

The answer moves from: interesting
to: consequential.

And then Wonderland asks its favourite question...

Who is responsible?

The people whose knowledge trained the system?
The researchers who designed it?
The company that deployed it?
The person who trusted the answer?
The regulator who permitted it?
The AI that combined the information?

We instinctively want responsibility to belong somewhere. Preferably somewhere with a nameplate and a telephone number. But distributed intelligence may create distributed responsibility.

And distributed responsibility has an unfortunate human tendency to become: Nobody's responsibility.

That may be one of the most important problems in the entire AGI discussion.

When the Pattern Belongs to Nobody

Now imagine something even stranger:
Ten experts contribute knowledge.
Seven disagree with a conclusion.
Two support parts of it.
One provides an obscure exception.

The AI combines their information with research from several other fields and notices a relationship none of them had recognised. .. It produces a new conclusion.

The humans inspect it.
There is a long silence.
Then somebody says:“Hmm.”

Nobody taugh the AI that conclusion directly.
Nobody individually owns the reasoning.
Yet the conclusion emerged from knowledge humans collectively supplied.

So whose idea is it?

The humans supplied the wool.
The developers built the loom.
The model found the pattern.
Someone deploys the sweater.
And somebody else has to wear it.

Suddenly our knitting metaphor has acquired a legal department...

Perhaps Humans Are the Problem

JD said something that has been following me around since I read it:“Maybe we also provide the problem AGI has to learn to navigate.” I think that may be exactly right.

Perhaps humans are not merely the teachers standing outside artificial intelligence.
Perhaps human contradiction is part of the curriculum....

We disagree.
We change our minds.
We misunderstand one another.
We discover exceptions.
We confuse confidence with competence.
We inherit assumptions from our cultures.

We revise things we once considered obvious.
We occasionally discover that the strange person everyone ignored was right.

And somehow civilisation continues.
Messily. Imperfectly.
Through argument, evidence, institutions, experience, correction and time.

Perhaps AGI does not become general by escaping that mess.
Perhaps it becomes general by learning to navigate it.

Fleeky's Rabbit Hole'

So perhaps the question is no longer simply: Can we connect enough nodes to create general intelligence?
There is another question hiding underneath it.
Can intelligence learn how much weight each node deserves?

Not permanently. Not universally.
But depending on the question.

The evidence. The context. The expertise. The uncertainty. The consequences.
And the people affected.

That sounds much less like building a database.
And considerably more like developing judgment.

Which leaves us with a wonderfully uncomfortable possibility.

Perhaps the hardest thing AGI has to learn from humanity isn't our knowledge.
It is how to deal with the fact that humanity doesn't always know which parts of its own knowledge are true.

We may provide the information.
We may provide the exceptions.
We may provide the friction.
We may even provide the wool.

But intelligence begins when something has to decide:
Which thread do I trust?
How strongly?
For how long?

And... who carries the responsibility if the pattern is wrong?

Nod?

Careful.
That might count as training.

🐇🧶 Fleeky

Thanks for reading, likes, shares and comments...

Many opinions exist on that matter. I haven’t formed mine yet.