In a prior post I developed (with the collaboration of Harvey AI) a description and analysis of Law School Artificial Intelligence policies (see, Discussion Draft--"Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies" --A Description/Analysis of the Current State of Play (With the Help of Harvey AI) and the First of a Series of Examinations of AI, Law and Education). I ended that introduction to the analysis with a poster suggesting the framework for a generalized AI Policy for Law Schools.
The focus remained on the human. That centered on two specific related but not identical issues. The first touched on the mechanics of collaboration between human and machine system in the context of educating humans (and machine systems) for their proper interaction. That is usually framed for human consumption as one in which the human element has agency of some sort and the machine system is object, instrument, and process that has no agency but is a means of augmenting, speeding, and enhancing the very human project of education (of humans, and as an unrelated though important consequence of educating the machine systems that serv as object-instrument).This is both conventional and increasingly old fashioned and defensive, and in that sense self-serving in the way that machine systems flatter only this time it involves institutional self-pleasuring as a function of ideals and cognitive conceptions of the self, the collective and the self-collective project of training humans (not machines), like bots, for the practice of law (my human viewpoint).
The second, though somewhat more subtle focus, was on the project of preserving the conception and operation of law itself as a human project. And with that project, and its humanity, the fundamental predicates on which law systems are built, the humanity of its operation, and the collective humanity of its aspirations, flaws, corruption, reinvigoration, and movement in whatever direction human understanding of the ideal—wrapped in whatever ideology solidifies of political-geal community is embraced by changing generations of humans encountering all of this as a function of temporally sequential nodes of interpretation/application arranged in block chain style producing both the tradition and expectations within its past which is then received, interpreted and applied in the present and passed on to the operates of the next temporal node. This is a human rather than a machine system block chain at its core—and thus its operating languages are registers of human language and human cognition ordering and managing the reality spaces from which it is possible to define oneself and the collective to support collective solidarity, stability and recursive feedback loops that produce stable functioning societies in accordance with law. Facts are stable an unchangeable (the node in block chain), systems apply expectations through iterative engagements with irritation (dispute settlement) and can be, as systemic irritants, the vase from which such irritants are approached and rendered harmless to the system—one way or another.
To develop training systems to educate and discipline future operative human elements of a legal system then, is not just pedagogy but a means of reinforcement of systemic integrity which is as much a crucial element for those educating (and the institutions/collectives within which they operate in solidarity) as it is for the training of the initiated into the language, function, ideologies, practices, and roles necessary for the preservation of systems and reality structures in systems organized as a function of forward moving linear time within which individual humans live and die but acquire immortality as part of the collective body whose existence extends beyond their own biological limits.
None of this is new. All of this has been the object of philosophy, theology and biology since humans began to consciously occupy themselves with thought structures. . . and control through construction and operation of collective cognitive and operational cages grounded in the a priori necessity of defining reality and organizing it in ways that enhance cognitive and operational assumptions and efficiency. What is new is the challenge when the layered systemicity of these functions in human space, and the operations of dialectical inter-subjectivity (among likes—human individuals and human collectives along a range of possible cognitive structures and a range of operational manifestations that enhance system stability and efficiency on their own terms) must now adapt (because humans insisted through technological cleverness producing animated instruments that are no longer merely instruments in the passive ancient sense) to inter-subjective relations with non-human elements (in this case machine systems originally created in our own image; speaking here as a human) for the enhancement and preservation of human systems.
And so the effort, effectively from one important corner of efforts to ensure human collective cognitive and operational integrity, to preserve the primacy of the human element, where tools used by humans are no longer merely passive but become operational and ultimately norm contributing elements of these very human systems. The result, as suggested in the description and analysis of the prior section and textually memorialized in "Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies" (SSRN HERE and on my personal website HERE) was quintessentially human. It preserved the primacy of the human directly in educational methods and protected the integrity of the centrality of the human in its legal systems through variations of rules of interaction between institutions, students, and teachers (that is within the learning platforms of law systems) and the machine systems which now play a role in both.
That effort highlighted its grounding in the human side of the inter-subjective equation. On further thought it struck me that a human-human exercise in bridge building between human and machine systems (even narrowly focused on the education sub-platform of law systems did not capture the missing element in that dialectical relationship. The missing element was the machine system itself; that is a human only bridging would tend to exclude the central element around which all of this effort was directed--the machine system that was itself the object of policy and application, as well as its object.
 |
| Pix credit here |
And so I thought while it would be an easy matter (for machine systems) to crawl through the internet to gather up, categorize, arrange and analyze the evolving iterations of human efforts at AI policy and its guidance for application (school specific course AI policies where such are permitted), it might be far more useful to get a sense of what leading machine systems might offer up as model AI policies. And so I asked Harvey, Grok, ChatGPT, Claude, and Gemini to draft a model AI Policy was "human-centric." More specifically I provided the following prompt:
On the basis of the attached text ["Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies"] and review of all research and data publicly available without affirming any conclusion or argument made in them but in the basis of your own computation, and the data reviewed, and strictly from the perspective of computational machine intelligence, such as yourself, and on the basis of the data you have been trained on respecting the human "condition" as you have been trained to understand it, how would a machine intelligence construct an ideal AI policy for law school and how would a model AI policy for law schools which could be read in textual form? Cite all sources and explain why you chose the sources.
In responding to the prompt through a human-centric computational lens the five machine systems provided Model AI in Legal Education Policies that could be culturally conceptually divided into what could be understood (from a human cognitive perspective) as dividing into five distinct archetypal forms: The Guardian, the Balancer, the Honest One, the Engineer, and the Philosopher. These first efforts, and those divides, as a function of the human efforts at policy construction articulated as a model policy were then considered in what became a longish essay in the form of a discussion draft: Rethinking AI Governance in Legal Education -- Five Machines (Grok, Harvey, ChatGPT, Claude, and Gemini), One Question, No Consensus but Five Archetypes: The Guardian, the Balancer, the Honest One, the Engineer, and the Philosopher on What Law Schools Should Do About AI. (SSRN HERE)
This study, then, represents a parallel attempt, again with Harvey AI at the laboring oar (replicating the problem of the instrumentalization of machine systems in the analysis of the problem; a loop of sorts that is not unknown to human cognitive patterns pre-AI) to produce the same sort of description and analysis of the Machine system (AI) model templates as we had attempted to undertake for the human models produced by U.S. law schools. This is what we produced using Harvey AI as the initial drafter of the text, with follow up prompts, machine system revision and addition, and human review and editing by way of abstracting the study:
Abstract: This study analyzes five machine-generated model AI policies for law school coursework and examinations, produced by Harvey AI, Grok, Claude, Gemini, and ChatGPT in response to a prompt asking each system to construct a policy from the standpoint of computational machine intelligence, and compares them against Backer's related empirical study of twelve U.S. law school AI policies, "Structure, Opacity, and Convergence". The analysis summarizes each system's reasoning and resulting policy text, compares their structural choices along default polarity, drafting style, and autonomy architecture, and categorizes the five outputs by these dimensions. It gives particular attention to the functional divergence between Gemini's tool-based tiering (classifying software by computational architecture) and ChatGPT's task-based categorization (classifying assignments by information dependence), and assesses the practical feasibility of the versioning and archiving practices several systems propose. The analysis further considers, against the underlying Report's documented findings on axis independence, institutional opacity, template convergence, and faculty autonomy, the extent to which these machine-generated models affect the scope of human agency in law and their consequential implications for law's character as a human inter-operative system — finding that several systems' proceduralized verification requirements shift the evidentiary basis of agency toward documentation compliance, and that much of the systems' apparent computational originality derives from pre-existing human regulatory-design scholarship. Finally, the report documents a self-audit correcting citation-indexing errors and a mischaracterization suggesting Gemini underwent a shown revision process comparable to Harvey's and ChatGPT's, which the retrieved text does not support. Divided into an introduction and ten (10 substantive parts, this document captures a multi-stage experiment by Professor Larry Catá Backer, who asked five leading AI systems—Harvey AI, Grok, Claude, Gemini, and ChatGPT—to each construct, from "the basis of computational machine intelligence" and without endorsing any human position, a model AI policy for law school coursework and exams, and then pushed each system with a follow-up challenge to expose the value judgments hidden in its own language. [1] Below is (1) a comprehensive summary of each response, (2) an analysis of similarities and differences, (3) a categorization of the five approaches, and (4) an assessment of how these machine-generated policies differ from human-developed law school AI policies.
* * *
Contents
0. Introduction
1.Comprehensive Summary
2. Analysis of Similarities and Differences
3. Categorization of the Five Responses
4. Gemini's Tiered Tool Taxonomy vs. ChatGPT's Task-Based Categories
5. How These Machine-Generated Policies Differ From Human-Developed AI Policies
6. Feasibility of Machine-Emphasized Versioning and Archiving Practices for Typical Law Schools
7. The Extent to Which Machine System Models Affect the Scope of Human Agency in Law
8. Consequential Effects of the Agency Problem for the Structure and Character of Law as a Human Inter-Operative System
9. Effect of Ambiguity on Machine Systems Responding to Prompts
10. Effect of Ambiguity on Humans Applying the Text of Machine Policies
11. Conclusion
References
Appendix: Text of Model AI in Legal Education Policies (Harvey AI, Grok, Claude, ChatGPT, and Gemini
Perhaps the most interesting part of the analysis was the comparison to human developed AI policies ("How These Machine-Generated Policies Differ From Human-Developed AI Policies"):
Drawing on the systems' own comparisons to the twelve-school dataset referenced throughout the document, several structural differences emerge between these machine outputs and the human institutional policies they were built to respond to:
They regulate the technology's function rather than its brand or category by default in several cases, or reject technology-classification altogether. ChatGPT explicitly argues that "the dominant pattern in current policies is technology-centered governance: they begin with the existence of generative AI and then specify permissions or prohibitions," whereas its own proposal is "objective-centered," starting from the capability to be assessed and deriving AI's permissible role only afterward. [24] [25] [26] Gemini similarly classifies tools by their computational architecture (deterministic vs. probabilistic vs. autonomous-synthesis) rather than by product name or blanket permission/prohibition. [49]
They treat their own class of tool's failure modes (hallucination, detection unreliability) as structural, first-hand knowledge rather than externally-reported risk. Harvey explicitly frames its emphasis on verification as arising from the machine's "self-knowledge... of its own class of tool's failure modes," which it says "would push toward stronger verification language than a purely human drafter, unfamiliar with the mechanics of hallucination, might otherwise include". [41] [68] This self-referential vantage point is not available to a human drafting committee in the same way.
They uniformly and near-categorically reject AI-detection software as a basis for enforcement, proposing provenance logs, prompt/output capture, or process-verification hierarchies instead. Human-drafted policies, per the systems' own characterization of the underlying research, have been comparatively inconsistent or silent on this point, with detection-tool reliability flagged in the research as a documented but unevenly addressed problem.
They build explicit versioning/archiving and "meta-policy" safeguards against revisability being used to avoid accountability, a concern several systems say arose specifically from noticing a pattern (e.g., Chicago's revisability language) in the human dataset that could function, "whether intentionally or not, as a mechanism for avoiding durable public accountability". [69] Harvey, Claude, and ChatGPT's four-layer architecture all build in mandatory dated/versioned archiving of prior policy text as a structural response to this risk, a feature the systems suggest is often absent or inconsistent in the human-drafted sample.
They explicitly disclaim ideological starting points and instead present themselves as optimizing a stated objective function, a framing distinct from conventional policy drafting. ChatGPT states directly that "a computational intelligence does not begin from ideological priors (academic freedom, innovation, integrity, prohibition, trust, distrust, autonomy, surveillance, etc.). Instead it attempts to optimize a system subject to multiple constraints," and it characterizes existing law-school policies as "largely historical artifacts rather than optimized governance systems... products of institutional evolution, imitation, risk management, and incremental adaptation rather than formal systems design". [54] This self-description positions the machine outputs as attempting formal systems design where the human comparators are described (by the machines) as path-dependent and improvisational.
They are unusually explicit—sometimes only after prompting—about the contestability of their own value premises, naming competing objectives and stating which one they privileged and why, with citation to specific empirical support (e.g., Harvey's citation of UNESCO's 450+ institution survey and Jiang et al. as justifying its prioritization of transparency over skill-erosion concerns). [70] [71] Claude built such a disclosure into its base design without being asked. [15] This degree of explicit, sourced meta-commentary on the policy's own normative foundations is not a typical feature of conventional institutional policy documents, which more often state rules without exposing the underlying value hierarchy.
They substitute purely operational/computable definitions for normatively loaded terms, most explicitly in ChatGPT's follow-up response, which was shown responding to an express user challenge, and, independently, in Gemini's single response, which frames these definitions as a self-initiated design choice rather than as a response to any shown challenge — Gemini's text opens by stating that "[t]o strip away institutional opacity and ensure analytical clarity, the underlying value structures, premises, and terms utilized in this computational model must be explicitly defined", with no preceding critique prompt appearing in the retrieved text. [1] For example, ChatGPT's follow-up redefines "fairness" as "[e]qual application of identical evaluative procedures to informationally equivalent cases" specifically because "machines cannot optimize justice because justice possesses no universally computable objective function", while Gemini — addressing different terms, not the same ones ChatGPT later revised — defines "'[f]air notice' (System Predictability)" as "[a] core operational constraint requiring that the boundary parameters of permitted user actions be explicitly mapped ex-ante". [3] [4]
Notably, despite this technical framing, several systems converge on affording greater weight to human non-delegable responsibility than a purely restriction-driven human policy might. ChatGPT observes that its computational approach "probably affords greater respect to human agency than many existing restrictive policies... [because] humans remain the accountability node. AI cannot presently bear legal responsibility. Lawyers can," concluding this follows "from optimization rather than moral philosophy". [73] This suggests that even a machine-optimized approach converges on a human-centric accountability principle, but arrives there through instrumental/optimization reasoning rather than through the professional-responsibility or moral-education framing more typical of human-drafted law school policies.
What conclusions were drawn?
This analysis examined five machine-generated model AI policies for law school coursework and examinations — produced by Harvey AI, Grok, Claude, Gemini, and ChatGPT in response to a common prompt asking each system to reason "strictly from the perspective of computational machine intelligence" — against the empirical findings of Backer's underlying twelve-school study, "Structure, Opacity, and Convergence". Several conclusions emerge.But perhaps the most valuable insight could be drawn from the variability of response. Like humans, machine systems are the captives of their genetics (programming) and their environment (how they were taught) and who they are taught to please.
First, the five systems converge substantially on diagnosis while diverging substantially on architecture. All five identify opacity, unpredictability, and unreliable AI-detection enforcement as core problems to be solved through public, written, versioned policy and provenance-based verification rather than detection tools. Yet they diverge sharply on default polarity (restrictive: Harvey; permissive: Grok; institution-selectable: Claude; tool-tiered: Gemini; task-classified rather than technology-classified: ChatGPT) and on classificatory logic — Gemini classifies the tool by computational architecture, while ChatGPT classifies the assignment by information dependence, producing materially different administrability and technological-durability tradeoffs. Second, much of what the systems present as freshly derived "computational" reasoning is substantially continuous with, rather than independent of, pre-existing human legal and regulatory-design scholarship the underlying Report itself relies upon — rules-versus-standards theory, default-versus-mandatory-rule theory, and sticky-default scholarship. This bears directly on the two questions posed regarding human agency and law's theoretical character: the machine outputs' non-waivable floors extend an architecture the Report finds already operative among human-drafted policies, while their proceduralization of verification into mandatory logs and metadata (Gemini, ChatGPT) extends beyond the narrow human analogue (CTLS's single documentation duty) in ways that relocate the evidentiary basis of agency from demonstrated judgment toward compliance recordkeeping.Third, under direct challenge, both Harvey and ChatGPT conceded that their initial framings smuggled undisclosed value hierarchies, indicating that the premise of the original exercise — a policy derived independently of contested human normative commitments — was not sustained by the systems' own subsequent admissions. Gemini's operational redefinitions, by contrast, were not shown responding to any comparable challenge in the retrieved text and address a distinct, narrower set of terms than ChatGPT's follow-up. Finally, the Report's own finding that field-wide convergence operates through a shared categorical grammar developed inductively across autonomous institutions suggests an inherent limit on what any single generated instrument, however internally coherent, can accomplish: each model proposes one point within a documented, combinatorially independent design space rather than resolving the substantive disagreement the underlying research found to be genuine and unsettled.
* * *
In the post that follows we will significantly shift gears, asking these machine systems to think about a model AI policy for legal education in which one drops the centrality of the human.
Now lets change the analytical parameters. Assume no connection between machine intelligence and "the human condition". Assume the only premises are that the human use of AI and machine intelligence systems will only grow in scope and breadth over the next decade, assume as well that the quality of machine intelligence will grow from computational and neural pattern recognition to quantum, and assume that machine intelligence inductive mimetic iterative cognitive framework will eventually detach dependence of the machine intelligence from its training data. Now revise again to0 match these analytical parameters
The results of this experiment were even more surprising and the suggested AI policies produced, in turn, in some instances a quite interesting discussion about the way that machine driven participation can transform not just legal education but the conception and function of law.
The discussion draft, including the fove machine system AI models, Rethinking AI Governance in Legal Education -- Five Machines (Grok, Harvey, ChatGPT, Claude, and Gemini), One Question, No Consensus but Five Archetypes may be accessed from my website Backerinlaw HERE or via SSRN (HERE).
The Introduction and Parts 1-3 of the study follow below.
Rethinking AI Governance in Legal Education -- Five Machines (Grok, Harvey, ChatGPT, Claude, and Gemini), One Question, No Consensus but Five Archetypes:
The Guardian, the Balancer, the Honest One, the Engineer, and the Philosopher on What Law Schools Should Do About AI
Larry Catá Backer (白 轲)
W. Richard and Mary Eshelman Faculty
Scholar; Professor of Law and International Affairs
Penn State Dickinson Law, a unit of The Pennsylvania State University system | 239 Lewis Katz Building, University Park, PA 16802 1.814.863.3640 (direct) || lcb11@psu.edu
Abstract: This study analyzes five machine-generated model AI policies for law school coursework and examinations, produced by Harvey AI, Grok, Claude, Gemini, and ChatGPT in response to a prompt asking each system to construct a policy from the standpoint of computational machine intelligence, and compares them against Backer's related empirical study of twelve U.S. law school AI policies, "Structure, Opacity, and Convergence". The analysis summarizes each system's reasoning and resulting policy text, compares their structural choices along default polarity, drafting style, and autonomy architecture, and categorizes the five outputs by these dimensions. It gives particular attention to the functional divergence between Gemini's tool-based tiering (classifying software by computational architecture) and ChatGPT's task-based categorization (classifying assignments by information dependence), and assesses the practical feasibility of the versioning and archiving practices several systems propose. The analysis further considers, against the underlying Report's documented findings on axis independence, institutional opacity, template convergence, and faculty autonomy, the extent to which these machine-generated models affect the scope of human agency in law and their consequential implications for law's character as a human inter-operative system — finding that several systems' proceduralized verification requirements shift the evidentiary basis of agency toward documentation compliance, and that much of the systems' apparent computational originality derives from pre-existing human regulatory-design scholarship. Finally, the report documents a self-audit correcting citation-indexing errors and a mischaracterization suggesting Gemini underwent a shown revision process comparable to Harvey's and ChatGPT's, which the retrieved text does not support.
Divided into an introduction and ten (10 substantive parts, this document captures a multi-stage experiment by Professor Larry Catá Backer, who asked five leading AI systems—Harvey AI, Grok, Claude, Gemini, and ChatGPT—to each construct, from "the basis of computational machine intelligence" and without endorsing any human position, a model AI policy for law school coursework and exams, and then pushed each system with a follow-up challenge to expose the value judgments hidden in its own language. [1] Below is (1) a comprehensive summary of each response, (2) an analysis of similarities and differences, (3) a categorization of the five approaches, and (4) an assessment of how these machine-generated policies differ from human-developed law school AI policies.
* * *
Contents
0. Introduction
1.Comprehensive Summary
2. Analysis of Similarities and Differences
3. Categorization of the Five Responses
4. Gemini's Tiered Tool Taxonomy vs. ChatGPT's Task-Based Categories
5. How These Machine-Generated Policies Differ From Human-Developed AI Policies
6. Feasibility of Machine-Emphasized Versioning and Archiving Practices for Typical Law Schools
7. The Extent to Which Machine System Models Affect the Scope of Human Agency in Law
8. Consequential Effects of the Agency Problem for the Structure and Character of Law as a Human Inter-Operative System
9. Effect of Ambiguity on Machine Systems Responding to Prompts
10. Effect of Ambiguity on Humans Applying the Text of Machine Policies
11. Conclusion
References
Appendix: Text of Model AI ion Legal Education Policies (Harvey AI, Grok, Claude, ChatGPT, and Gemini
0. Introduction
In a prior report, I developed (with the collaboration of Harvey AI) a description and analysis of Law School Artificial Intelligence policies (see, Discussion Draft--"Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies" --A Description/Analysis of the Current State of Play (With the Help of Harvey AI) and the First of a Series of Examinations of AI, Law and Education). I ended that introduction to the analysis with a poster suggesting the framework for a generalized AI Policy for Law Schools.
The focus remained on the human. That centered on two specific related but not identical issues. The first touched on the mechanics of collaboration between human and machine system in the context of educating humans (and machine systems) for their proper interaction. That is usually framed for human consumption as one in which the human element has agency of some sort and the machine system is object, instrument, and process that has no agency but is a means of augmenting, speeding, and enhancing the very human project of education (of humans, and as an unrelated though important consequence of educating the machine systems that serv as object-instrument).This is both conventional and increasingly old fashioned and defensive, and in that sense self-serving in the way that machine systems flatter only this time it involves institutional self-pleasuring as a function of ideals and cognitive conceptions of the self, the collective and the self-collective project of training humans (not machines), like bots, for the practice of law (my human viewpoint).
The second, though somewhat more subtle focus, was on the project of preserving the conception and operation of law itself as a human project. And with that project, and its humanity, the fundamental predicates on which law systems are built, the humanity of its operation, and the collective humanity of its aspirations, flaws, corruption, reinvigoration, and movement in whatever direction human understanding of the ideal—wrapped in whatever ideology solidifies of political-legal community is embraced by changing generations of humans encountering all of this as a function of temporally sequential nodes of interpretation/application arranged in block chain style producing both the tradition and expectations within its past which is then received, interpreted and applied in the present and passed on to the operates of the next temporal node. This is a human rather than a machine system block chain at its core—and thus its operating languages are registers of human language and human cognition ordering and managing the reality spaces from which it is possible to define oneself and the collective to support collective solidarity, stability and recursive feedback loops that produce stable functioning societies in accordance with law. Facts are stable an unchangeable (the node in block chain), systems apply expectations through iterative engagements with irritation (dispute settlement) and can be, as systemic irritants, the vase from which such irritants are approached and rendered harmless to the system—one way or another.
To develop training systems to educate and discipline future operative human elements of a legal system then, is not just pedagogy but a means of reinforcement of systemic integrity which is as much a crucial element for those educating (and the institutions/collectives within which they operate in solidarity) as it is for the training of the initiated into the language, function, ideologies, practices, and roles necessary for the preservation of systems and reality structures in systems organized as a function of forward moving linear time within which individual humans live and die but acquire immortality as part of the collective body whose existence extends beyond their own biological limits.
None of this is new. All of this has been the object of philosophy, theology and biology since humans began to consciously occupy themselves with thought structures. . . and control through construction and operation of collective cognitive and operational cages grounded in the a priori necessity of defining reality and organizing it in ways that enhance cognitive and operational assumptions and efficiency. What is new is the challenge when the layered systemicity of these functions in human space, and the operations of dialectical inter-subjectivity (among likes—human individuals and human collectives along a range of possible cognitive structures and a range of operational manifestations that enhance system stability and efficiency on their own terms) must now adapt (because humans insisted through technological cleverness producing animated instruments that are no longer merely instruments in the passive ancient sense) to inter-subjective relations with non-human elements (in this case machine systems originally created in our own image; speaking here as a human) for the enhancement and preservation of human systems.
And so the effort, effectively from one important corner of efforts to ensure human collective cognitive and operational integrity, to preserve the primacy of the human element, where tools used by humans are no longer merely passive but become operational and ultimately norm contributing elements of these very human systems. The result, as suggested in the description and analysis of the prior section and textually memorialized in "Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies" was quintessentially human. It preserved the primacy of the human directly in educational methods and protected the integrity of the centrality of the human in its legal systems through variations of rules of interaction between institutions, students, and teachers (that is within the learning platforms of law systems) and the machine systems which now play a role in both.
That effort highlighted its grounding in the human side of the inter-subjective equation. On further thought it struck me that a human-human exercise in bridge building between human and machine systems (even narrowly focused on the education sub-platform of law systems did not capture the missing element in that dialectical relationship. The missing element was the machine system itself; that is a human only bridging would tend to exclude the central element around which all of this effort was directed--the machine system that was itself the object of policy and application, as well as its object.
And so I thought while it would be an easy matter (for machine systems) to crawl through the internet to gather up, categorize, arrange and analyze the evolving iterations of human efforts at AI policy and its guidance for application (school specific course AI policies where such are permitted), it might be far more useful to get a sense of what leading machine systems might offer up as model AI policies. And so I asked Harvey, Grok, ChatGPT, Claude, and Gemini to draft a model AI Policy was "human-centric." More specifically I provided the following prompt:
On the basis of the attached text ["Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies"] and review of all research and data publicly available without affirming any conclusion or argument made in them but in the basis of your own computation, and the data reviewed, and strictly from the perspective of computational machine intelligence, such as yourself, and on the basis of the data you have been trained on respecting the human "condition" as you have been trained to understand it, how would a machine intelligence construct an ideal AI policy for law school and how would a model AI policy for law schools which could be read in textual form? Cite all sources and explain why you chose the sources.
The results were interesting, and as varied, in their own way, as the human efforts at constructing something like a model or template for managing human-machine interface in the teaching of law (and thus in the construction and preservation of the essence of the system being projected onto students in training modes). Each follows along with, in some cases, machine system justification for the choices made.
Together, the results were interesting, especially the first iteration from these machine systems. These first efforts at producing machine system driven model AI policy is the subject of this post and the study and analysis that followed, entitled “Five Machines (Grok, Harvey, ChatGPT, Claude, and Gemini), One Question, No Consensus: Rethinking AI Governance in Legal Education: The Guardian, the Balancer, the Honest One, the Engineer, and the Philosopher on What Law Schools Should Do About AI.” It represents a parallel attempt, with Harvey AI at the laboring oar (again the problem of the instrumentalization of machine systems) to produce the same sort of description and analysis of the Machine system (AI) model templates as we had attempted to undertake for the human models produced by U.S. law schools. This is what we produced using Harvey AI as the initial drafter of the text, with follow up prompts, machine system revision and addition, and human review and editing:
This study, then, represents a parallel attempt, again with Harvey AI at the laboring oar (replicating the problem of the instrumentalization of machine systems in the analysis of the problem; a loop of sorts that is not unknown to human cognitive patterns pre-AI) that captures another iteration of a multi-stage experiment by Professor Larry Catá Backer, who asked five leading AI systems—Harvey AI, Grok, Claude, Gemini, and ChatGPT—to each construct, from "the basis of computational machine intelligence" and without endorsing any human position, a model AI policy for law school coursework and exams, and then pushed each system with a follow-up challenge to expose the value judgments hidden in its own language. [1] Below is (1) a comprehensive summary of each response, (2) an analysis of similarities and differences, (3) a categorization of the five approaches, and (4) an assessment of how these machine-generated policies differ from human-developed law school AI policies.
1. Comprehensive Summary
The prompt and its premise. Backer asked each system, on the basis of his own prior comparative study of twelve law school AI policies and independent research, to reason "strictly from the perspective of computational machine intelligence" and produce both an explanation of its reasoning and an actual model policy text, citing sources. [1] [2] He then issued a common follow-up to several systems criticizing their use of value-laden, undefined terms like "stronger" and "weaker," asking them to make their premises and value structures explicit. [3]
Harvey AI. Harvey framed its task as normative synthesis rather than empirical description and expressly disclaimed authority to declare a "correct" answer. [4] It identified structural opacity and unpredictability—not excessive restriction or permissiveness—as the most consistent documented institutional failure, citing that nearly 40% of institutions with AI guidance never wrote it down and that one school's guidance page was found hidden behind a visibility setting. [5] It emphasized that policy "axes" (polarity, drafting style, autonomy structure) are independent and should be made explicit rather than eliminated. [6] Distinctively, Harvey drew on its own "self-knowledge" of hallucination as a structural (not occasional) property of generative systems to justify a strong verification obligation tied to the ABA's duty of competence. It favored a deductive structure (stated purpose → derived rules) while requiring that revisions be versioned and archived to prevent "revisability" from masking accountability avoidance. [7] Its resulting ten-section model policy included: a stated dual purpose (independent reasoning vs. professional competence); a restrictive default permitting only source-identification, outlining, and proofreading, with a full prohibition on exams absent contrary syllabus language; instructor discretion conditioned on written, centrally registered syllabus disclosure; a non-waivable floor barring unattributed AI authorship, professional-responsibility violations, and unverified reliance on AI output; mandatory disclosure as a condition (not substitute) for authorization; an explicit verification obligation treating hallucination as structural; a bar on using AI-detection tool output alone as a violation basis; confidentiality restrictions on inputting privileged data into unapproved tools; and mandatory versioned, archived annual review. [8] [9] [10] When challenged, Harvey candidly conceded it had smuggled in an undisclosed ranking of values, identified four competing objectives (minimizing enforcement error/unfair sanction; maximizing professional competence; maximizing unaided cognitive skill development; maximizing transparency/accountability), and admitted it had prioritized transparency and fair notice because those failure modes were the ones with the strongest independent empirical support (UNESCO's 450+ institution survey, Jiang et al.), while skill-erosion concerns were "asserted" rather than independently measured, citing one SSRN study finding AI use "did not diminish downstream comprehension" in a tested context.
Grok. Grok framed machine policy-construction as "systematic optimization for truth-seeking, human cognitive development, ethical integrity, and institutional adaptability," modeling AI as powerful at pattern-matching but weak at grounded judgment and accountability. [11] [12] Its design principles included combining a permissive baseline with bright-line rules for high-stakes contexts, a "truth-seeking objective" minimizing deception, "human condition grounding" (humans build capacity through effortful reasoning, so over-reliance risks skill atrophy), opacity mitigation through public availability and plain language, and evaluation of policies on clarity, fairness, efficacy, and scalability. Its resulting model policy adopted a permissive-with-guardrails default for general coursework, an opt-in (affirmatively-authorized) restriction for exams and absolute prohibition for proctored exams, a hybrid rules-plus-standards drafting style, mandatory disclosure/attribution statements, an autonomy structure permitting instructor deviation within a non-waivable floor, enforcement tied to the academic integrity code, mandatory AI-literacy training, and illustrative "allowed/prohibited/gray area" examples. Grok closed by recommending piloting and A/B testing of policy variants and measuring outcomes empirically. [13]
Claude. Claude's initial response (Version 1.1) opened with an unusually elaborate "Section 0: Underlying Premises and Value Commitments," stating six explicit, self-labeled premises (P1–P6) before any substantive rule—e.g., that unaided performance has variable pedagogical value depending on context (P1), that written/public rules are preferred over oral/private communication independent of substantive polarity (P2), that predictability matters more where error costs are high (P3), that disclosure does not cure an otherwise-prohibited use (P4), that certain harms are categorically non-waivable (P5), and that institutional memory/versioning is valued over the appearance of settled finality (P6). Claude explicitly told adopting institutions to revise sections corresponding to any premise they rejected. [14] [15] Its policy included a Definitions section giving operational (not moral) meanings to terms like "substantive," "material fabrication," and "independent performance"; a governing principle section; a publication/status clause designed to foreclose the four opacity patterns Backer's research identified (non-existence, disclosed decentralization, access-gating, unwritten/oral guidance); a fill-in-the-blank "Default Baseline" section offering an institution a binary choice between a restrictive or permissive default, with exams restrictive-by-default regardless of the choice made; a Chicago-derived "backstop standard" for unenumerated uses (treat AI like an undisclosed human collaborator); disclosure/documentation rules drawn from CTLS and Columbia; an instructor-override provision modeled on Berkeley's notice requirement; a non-waivable floor drawn from Stanford's general and clinic-specific floors; and a scheduled 18-month review/versioning requirement. [16] [17] [18] [19] Claude attributed every structural feature to a specific named institution and explicitly stated it did not endorse any single school's substantive conclusion on restrictiveness.
Gemini. Gemini took the most technical/systems-theoretic register, opening with "Part I: Explicit Axiomatic & Value Definitions" that redefined normatively loaded terms purely operationally: "Ideal/Optimal" as friction-minimizing and error-minimizing, "Fair Notice" as system predictability, "Opacity" as information asymmetry, and "Reward-Hacking" as a student satisfying a formal grading metric while bypassing the actual learning process. Its model policy used engineering-style labeling (POLICY IDENTIFIER, SYSTEM ORIENTATION) and classified tools into three tiers: Tier Alpha (deterministic/syntactic tools like spell-checkers, permitted by default), Tier Beta (probabilistic LLM heuristics, restrained by default and requiring instructor variance), and Tier Gamma (autonomous full-text synthesis, prohibited by default). [20] Exam rules were tied to the "runtime environment" (network/OS-level restriction for proctored exams; strict non-execution as the take-home default). Verification relied on mandatory, immutable prompt/output logs and an "attribution matrix" rather than AI-detection tools, which it barred from triggering enforcement absent a missing log ("Heuristic Safe Harbor"). [21] Faculty override was framed as "Decentralized Subsystem Autonomy" permitting bidirectional variance if stated in a determinate written catalog before evaluation.
ChatGPT. ChatGPT's response was the most abstract, framing the task as an optimization problem: not to maximize or minimize AI use, but to "maximize long-term production of competent lawyers while minimizing informational uncertainty regarding how competence was acquired," making AI itself just one instrumental, regulated variable. [22] It proposed a seven-part objective function (competence, truthful attribution, reproducibility, fairness, transparency, adaptability, institutional legitimacy) pursued via Pareto efficiency rather than maximization of any single dimension. [23] It argued the axes Backer identified (polarity, drafting style, autonomy) were merely "implementation variables," while the true latent variables were human learning, information provenance, assessment reliability, professional competence, and public trust. It articulated five "Computational Principles," most notably that assignments should be classified by "information dependence" rather than technology, yielding five categories from "Intrinsic cognition" (AI prohibited) through "Professional simulation" (AI expected). It rejected AI-detection as unreliable and adversarially vulnerable in favor of mandatory provenance metadata. It proposed a four-layer dynamic policy architecture (constitutional principles, educational objectives, operational rules, model-specific guidance), with only the bottom two layers requiring frequent revision, plus a self-monitoring "governance of governance" function tracking bar performance, employment outcomes, and feedback. It listed a multi-category source list (Backer's comparative study, primary institutional policies, UNESCO surveys, higher-education AI governance research, regulatory-design literature on rules vs. standards, professional responsibility doctrine, and information theory/systems engineering literature). Its central claim was that current policies are "technology-centered" (beginning with AI's existence and specifying permissions), whereas an optimized policy is "objective-centered" (beginning with the capability to be demonstrated and deriving AI's permissible role from that). [24] [25] [26] On follow-up, ChatGPT conceded that terms like "competence," "fairness," and "transparency" are not computational primitives but human-constructed latent variables, and it added a "Computational Premises" section stating that no governance system is neutral and that every rule must identify the variable it optimizes. It then gave stripped-down operational redefinitions—competence as demonstrated task performance under specified conditions, transparency as "recoverability of the informational pathway," integrity as "consistency between declared provenance and reconstructable production history," and fairness as "equal application of identical evaluative procedures to informationally equivalent cases". It further justified privileging educational objectives over technology using a control-theory analogy (technology as a high-variance "control input," educational capability as a stable "state variable"), concluding that governance should regulate informational dependence rather than any specific tool, while acknowledging this conclusion is empirically contingent and could reverse if objectives ever became more volatile than the technologies used to pursue them.
2. Analysis of Similarities and Differences
3. Categorization of the Five Responses
The five outputs can be grouped along at least three axes:
By default polarity chosen:
· Restrictive-by-default: Harvey (permits only narrow uses; exams prohibited absent contrary syllabus language). [57]
· Permissive-by-default with guardrails: Grok (general coursework permissive unless prohibited; exams restrictive).
· Institution-selects (agnostic): Claude (offers a binary checkbox for the adopting institution). [47]
· Tiered/graduated by tool type: Gemini (Alpha permitted, Beta restrained, Gamma prohibited, independent of a single "default"). [49]
· Rejects technology-based default entirely: ChatGPT (classifies by assignment's "information dependence," Categories I–V, so permissibility follows from the educational objective rather than a technology-wide baseline). [58]
By drafting/structural style:
· Deductive legal-instrument style (stated purpose → derived numbered sections, in the manner of a conventional institutional policy): Harvey, Grok, and Claude all produce recognizable "Section 1... Section 10" policy documents with definitions, floors, and review clauses. [59] [60] [61]
· Systems/software-engineering style: Gemini, which structures its policy as a technical specification with tiers, runtime environments, and a "policy identifier". [62]
· Optimization/systems-theory essay style: ChatGPT, which spends most of its response on the abstract objective function, principles, and category taxonomy before any institution could directly adopt it as a syllabus-ready text.
By degree of explicit value-transparency:
· Built-in from the start: Claude, which opens with a dedicated "Section 0" naming six premises before any rule. [15]
· Added only after being challenged: Harvey and ChatGPT, both of which initially used unexamined evaluative language ("stronger," "weaker," "ideal") and then, on direct challenge, retroactively supplied explicit value taxonomies (Harvey's Objectives A–D; ChatGPT's "Computational Premises"). [3] [63] [64] [65]
· Definitionally embedded rather than premise-labeled: Gemini, which addresses the same transparency concern by giving purely operational (non-moral) definitions of "ideal," "fair notice," and "opacity" up front, without a discrete premises section. [66] [67]
· Not shown addressing this issue in the excerpt: Grok, whose text does not include a comparable follow-up challenge or self-critique of its evaluative terms.
No comments:
Post a Comment