This is the first in a series of essays in which I try to think through the question of what assumptions social institutions make about the capacities of the human agents who constitute them.
If I am a market or an organization or a state, what capacities and limitations do I expect the human agents of whom I am composed to possess? The motivation for asking this question is so that we can ask whether and how these institutions, as we currently know them, are robust to the presence of super-intelligent agents.
The Jury
Consider the basic idea of a group appointing a subset of its members to consider a question and render a decision that will be accepted as the group’s answer to the question.
Let’s think of a jury as a simple human social institution. It’s “institutional” insofar as members of a group see a jury as one (perhaps among others) method for accomplishing a task together. It’s a social organizational recipe at hand within the group. Members recognize the kind of situations where a jury might be useful and they have a shared sense of how to set one up, how it should proceed, and what the status of its results will be.
Next we can ask what capacities of human agents are implicit in the idea of using a jury to answer a question for a group.
We seem to be assuming:
- basic cognitive processing: the agents can understand the context and the question and retain information
- capacity for judgment: the agents can form an individual answer to the question, perhaps based on information presented
- autonomy
- ability to communicate
- capacity to engage in a group process, recognize rules or methods
- recognize role of juror and make a commitment to fulfilling the duty
Alongside the minimum affirmative capacities the idea of a jury assumes, there are also implicit “maximal” capacities. When we constitute a jury as a decision-making process or a truth determining process or a verdict determining process, we are probably assuming the agents selected for the jury have certain limits such as:
- non-expertise
- non-omniscience
- cognitive limits comparable to other members
- cognitive biases
- depth of consideration
- interplay of rationality and everyday reasoning
- limits to speed of thinking
Would a jury work if one or more of its members were agents that exceed these limits? A super-intelligent juror might be too hyper-rational for others, unable to overlook details, incapable of overlooking some variables or ruling out of bounds some hypotheticals, to reach “common sense” results. It might not be capable of aligning with an idea like a (mere) “reasonable person” standard. A super-intelligent juror might be difficult for ordinary intelligences to deliberate with or it could undermine the benefits of diversity of perspectives that a jury provides. This juror might have access to too many different moral/ethical paradigms to be able to align with the values of the group. The presence of a super intelligent juror might force a shift from an institution where truth is expected to emerge from an averaging of peers to one where truth emerges from the mouth of an oracle or judge.
This is just a preliminary exercise to sketch the way these analyses might proceed. We start by identifying a social institution. We then ask what assumptions about human agents does it make. Initially we ask what minimal capacities of agents does the institution require and then we ask what maximal capacities might the institution as we know it depend on. Then we articulate the ways in which the participation of a super intelligent agent might distort the institution.
Innoeduvation 