I have been "stress testing" an application of a model AI
Classroom policy that I developed over the summer 2026 (see "AI assists.
You think. You analyze. You write. You take responsibility": Creating a
Course AI Use Policy Template --Policy Text, Justification and Rule Summary for
My Law & Religion Class at Penn State Dickinson). This is not a test in the classical sense but an effort to draw insight from attempting something new in class. That, in turn, is no different from the sort of self analysis I undertake with each class. The object was not, to be clear, to design a test in the form of a class, but to draw from class experiences insights on how to make the class better and what the state of knowledge is against which such improvement could be made--and then to share these insights--personal, anecdotal and generalized. It is in that sense that one might loosely identify the objects from out of which these observations and self assessments could be focused on the key stakeholders, colloquially our "test
subjects": (1) the students in my class, one a large and traditional
advanced basic course in corporations, the other a small class on advanced
issues in the Constitutional law of religion; (2) me, the faculty member who is
meant to model, apply, oversee, and ensure positive outcomes, even under
conditions of learning "struggle"; (3) the institutional capacity to
support and absorb stress, failure, success and to be nimble in the face of
changing conditions. It is a stress test loosely in the traditional sense — a
form of deliberately intense or thorough testing, used to determine the
stability of a given system, critical infrastructure or entity — though hardly
a proper scientific test at this stage, where the object is to figure out the
terrain and the points where more finely tuned and proper scientific study may
be attempted. The experience is ongoing but a number of insights begin to take
form:
1. Students, collectively, may not be ready. I overestimated student knowledge of, experience with, and readiness
to attempt to integrate AI and tech based products into their work. And of
course, the range of student experience with, desire to use, and understanding
of tech in general and AI in particular, as applied in a classroom setting is
profoundly varied. Even if they have experience, they have no working
knowledge, for the most part, in translating general experiences with AI and
other tech based modalities into the specifics of AI and other tech based use
in legal work, and more specifically, in legal work within an educational
institution the quirks of which may be different than those of legal practice,
government, enterprise or other work. A substantial amount of empirical work is
required to get a better sense of the state of student knowledge and experience
to better refine the embedding of tech and AI based knowledge production and
dissemination into a classroom setting. And even more work is required to
understand how that process is being evolved with courts, administrative
agencies, and law firms. The first step might include developing polling data
on current students and incoming students to get a better sense of
institution-specific characteristics of student populations, from which policy
and action may be better aligned with realities.
2. Students are
remarkably resilient. For all of that, the first several weeks of class —
the most stressful part of this experiment on student capacity — went
surprisingly well. It went remarkably well despite any number of glitches on my
part, and the need for students to develop new talents and adjust to new forms
of knowledge dissemination and production for which they had little experience
to draw on from their years of schooling. It is true enough that students
struggled, but they struggled in a positive way (even if they might not have thought
so). A few preliminary observations, though, are worth making:
a. Front loading
actual AI work in the beginning of the semester is not a good idea. It was the
product of my overestimation of student experience with and comfort around AI
and other tech based systems. That was a mistake. A better approach would be to
first produce a series of training exercises, in the manner of the training for
BOTS, and then to use that as the base for the blending of tech and normative
learning. That is a project I am now starting to undertake, but I suspect it
will produce its own challenges. There is another reason as well. I found, at
least this term, that students are risk averse. This is not a criticism: within
learning cultures in which everything is assessed and in which assessments are
critical to sorting processes for student insertion in wage labor markets for
which educational institutions play a part, students will tend to assess the
value of a course at least in part as a function of the risk of adverse
assessment, which is in turn assessed as a function of their assessment of
their capabilities and the time and interest they might have given a particular
set of risks. I have no quarrel with that. However, given the realization of
the amount and character of work now expected of them (moving from the
traditional passive to a far more active role in learning) and the need for
assessment as a function of the add/drop period, I should have assumed a
substantial probability of substantial movement in but mostly out of a course
the risk of which was hard to assess but likely higher than traditional
courses. The resulting movements caused substantial problems with group
assignments and combined with front loaded group work augmented challenges for
students and for the instructor as well.
b. Student
learning cultures must be reframed with sensitivity to the depth and
naturalization of inculcated educational processes and expectations which are
now naturalized within student cognitive "expectations" of a
"learning" and "classroom" experience. That requires a
broader conversation which will be difficult and ultimately a slow and likely
reluctant change in such expectations from faculty and educational
institutions. That is likely to take at least a decade given the cultural
conservatism of institutional sectors, and the current system grounded in a
sort of computational sycophancy system in which processes are produced as a
mutual accommodation of student desires and educational path dependent
processes.
c. Students prefer
substantial guidance, especially at the beginning of a course. Part of the
challenges that emerged with the course design was the risk I took in combining
two discrete changes. The first was of course the introduction of a heavy tech
component. The second was a more robust move toward student driven learning,
something that I had been working toward slowly over the last decade. This year
I added greater student learning autonomy guided through the production of very
detailed student teaching notes that I prepared in lieu of a lecture and that
covered virtually all of the contents of a class lecture drawn from the
materials, and a set of Infographics that provided a visual aid to the learning
to be undertaken for each class. That appears not to have been sufficient; at
least for the first third of the term. My sense is that even if the instructor
adds little more to the materials, their presence appears to be a critical
performative element of student learning, at least in the initial part of a
semester.
With this in mind I will now try an alternative and consider its value both for pedagogy and effective student engagement. I will stretch each group assignment into two classes. For the first of the classes student groups will meet and work on their presentations, including the background papers and PowerPoint, During class my TA or I will walk around , observe, and help. Student groups will then make their oral presentations on the following day. The down side is that it consumes a bit of class time. The benefit, to be tested, is that it will add an incrementally significant element to learning,
d. Grading can be
an impediment to learning and certainly to risk taking. By this I do not mean
course grading; I mean the assessment of the activities of students now
fashionably broken down into smaller bits each of which serves as an evaluation
layer that have become a central focus of student strategic behaviors. Not that
I blame them; they are rational actors; and not that faculty were foolish, the
intent was to enhance learning and reduce the stress of a single assessment. My
mistake was to grade the exercises under conditions of heightened risk
(especially of the unknown) — that is of risk with respect to which prior
student experience offered little by way of assessing risk. I will now no
longer grade the initial exercises — they are a learning experience rather than
an assessment exercise. I will evaluate them as a learning tool.
The deeper question — what constitutes valid assessment when
AI is embedded in the production process — is not addressed. My policy
template's certification requirement "converts the disclosure obligations
into an affirmative representation," but the stress test does not discuss
whether students found the certification process intelligible, whether the
appendix requirements were practicable, or whether the 100-word / 10% / 30%
numerical ceilings functioned as intended in practice. This is something I will
consider further with students.
e. Student equity
issues. The policy template developed for the courses (see "AI assists.
You think. You analyze. You write. You take responsibility": Creating a
Course AI Use Policy Template --Policy Text, Justification and Rule Summary for
My Law & Religion Class at Penn State Dickinson) explicitly
flags equity concerns — "students with more experience prompting AI tools,
or with access to premium tools, could gain a differential advantage" —
and frames disclosure as "a necessary first step" in exposing that
differential. The stress test appears to confirm a "profoundly
varied" range of student experience. It may be necessary to consider a
more robust approach to these disparities. Two issues merit further study:
whether equity issues ought to affect consideration of the role of evaluation
rather than grading at least in the initial projects, and whether there is a
way to more precisely measure differentials other than qualitatively. That
discussion, in turn, may be a function of the goal of embedding AI in
coursework, and in that connection whether this suggests a stronger case for
developing an AI and tech based skills/practice curriculum.
3. Faculty, collectively, may not be ready ready. Let me raise several general classes of issues preliminarily:
a. Realistic faculty
baselines. I underestimated the role of faculty in guiding AI use and
miscalculated the guidance necessary to embed AI in student work. More
interesting, for me, was a failure to note that the connection between
technology and knowledge production would be readily apparent and easy to
manifest in faculty and student work. I also underestimated the amount of
learning that is required to fully develop an integrated set of teaching
pedagogy that integrates AI learning with the traditional flows and substantive
knowledge dissemination expectations of students. Moreover, faculty mirror
students in both experience with AI, and a willingness or interest in using AI.
b. The effects of tech
and AI on not what we teach but how we understand the objects of instruction.
As important, faculty like many law firms with whose members I have spoken, are
at once aware that technology changes the relationship of the human to
knowledge generation and production, as well as knowledge dissemination, but at
the same time no one of my acquaintance has yet to claim a firm grasp of the
way that tech, and especially AI tech, actually changes the relationship
between what it means to produce knowledge, to curate knowledge production, to
deploy knowledge nor the way that human machine interactions changes the
relationship between knowledge production and the role of humans in it. Many of
the people of my acquaintance hold fast to the ideals of the pre-tech age in
those regards. It is not clear that those expectations, now pretensions, will
be relevant much longer. But the alternatives are hard to describe much less to
analyze in a useful way.
c. Consequences for
any "Model AI in Class Policy" template. My Model Policy Template
is built around a "narrow, conditional exception layered onto a default of
independent work, with permission calibrated to where oversight and iterative
disclosure are feasible." The stress test suggests that students may lack
the baseline AI literacy to meaningfully comply with disclosure and appendix
requirements (Section 1), and that front-loading AI work was premature (Section
2a). It may follow that it is necessary to circle back to ask: what does this mean for the policy template
itself? Does the template need a phased implementation protocol? Should the
disclosure-and-appendix architecture be preceded by a training module? My
template acknowledges it is "designed to be tested with students in an
actual course setting and refined based on what that experience reveals,"
but the stress test also suggests that further refinement may be required to
make the policy more relevant or at least connected to student realities, at
least as they exist now.
4. Institutions, as a collective, may not have built capacity to be ready. That produces an institutional issue. Might it be necessary to
develop initial evaluation of student knowledge of AI systems and use; if that
is the case does that suggest the need for separate AI specific training,
perhaps as part of a first-year curriculum and as an augmentation of the
traditional courses in legal writing — now legal writing and tech? These are
questions we have not yet even begun to ask. At a higher level of institutional
ordering one might also have to ask about differentiation between U.S. and
non-U.S. students in skill sets and more importantly in needs and expectations
in home countries. Indeed, this is no small matter, and a complicated one. It
might be necessary to develop at least some sensitivity to the issue, and it
also might suggest that AI skills courses would have to be developed with this
fundamental differentiation in mind. On the other hand foreign students seeking
grounding in U.S. approaches might be most well served by deep integration into
U.S. approaches and expectations in the utilization of AI tech. The same set of
considerations may apply to faculty who might require evaluation and training.
But here the institutional environment becomes more complicated because one has
yet to confront the issue of AI use as a function of academic freedom.
5. Education cultures
make the introduction of AI based teaching and knowledge
dissemination/production more difficult. My prior work suggests that while
the general population of stakeholders in law schools may be progressive in
matters of politics and culture, they appear to be quite conservative and
traditionalist, one might be tempted to say reactionary, in the face of the
potentially sweeping changes that technology, and especially AI based tech, may
bring not just to the practices of education but also to its fundamental
pedagogies and working styles. The current trend, marked by wariness and a
muscular effort to preserve the present against the possibly transformative (or
corrupting) potential that tech in general and AI in particular brings, is
likely to remain dominant in the short term and then necessarily change as
judicial and legal or attorney practices (and expectations) change. But that
requires ongoing faculty discussion and before that perhaps some "bringing
up to speed" exercises (and also for students). These in turn will clarify
and more precisely describe the choices going forward.
6. AI adds another
level of stress for teaching and learning that may produce both student and
faculty resistance. AI adds a layer of uncertainty — both in terms of
substantive knowledge and in process expectations — that increases risk, and
thus student stress, in an environment in which every response is both a test
and an evaluation. In evaluation orderings, like law school, where every
assessment can contribute to micro movements in relative performance status
which affects student understanding of successful insertion in legal labor
markets, the addition of a great unknown, AI, as a substantive field of learning
and as a process embedded in learning a traditional field of knowledge, can add
stress. That stress has two parts — the first is in the mastery of the subject,
the second is in the ability to properly apply it under conditions of
substantial uncertainty.
7. It is not yet clear
what is the nature and role and situating of AI and other tech based innovation
with effects on legal practice. That is particularly, and perhaps acutely,
the case with respect to AI in relation to fields of knowledge and
student/faculty roles. Legal tech could be described in good faith as its own
sub field. That, in turn, would suggest that legal tech might be best suited,
at least in the middle term, as its own course, as a separable object of study.
Legal tech, on the other hand, might be understood as instruments available for
legal study and in that sense ought to be part of the pedagogy of normative
courses and clinicals, each modified to suit the field of study. It is likely
more reasonable to apply the legal writing analogy to tech — that is it is both
its own field of study and deeply embedded in substantive and clinical courses.
That may, in turn, require substantial curricular changes.
8. Disaggregating
Issues. As an initial effort to disaggregate the issues around tech and
tech based education I have attempted an initial categorization of issues as
follows:
a. Students. This
can be subdivided into three sub categories.
The first is external — managing the environment in which
students may use, and learn to use, tech based and AI mechanisms in the
production, dissemination and analysis of legal issues relevant to their
courses and ultimately to their work as lawyers. To that effect the traditional
focus on classroom model AI policies and more generally law school or
institutional guardrails for use of AI and other tech by students is a central
element of contemporary management.
The second is internal — training students in the use of
tech based and specifically AI grounded "tools" (the word is in
quotes precisely because it is not clear that AI is merely a passive tool. Nor
is it clear that AI will be treated merely as an instrument for much longer).
The two are intimately connected but not yet in ways that it
is clear we understand well enough to act, even in the short term. Still the
dialectical relationship provides a basis for developing plans and modalities
for advancing student learning and student learning environment well enough.
The third is the cultivation and management of student
training, expectations, risk assessment, and ultimately of their baseline
engagement with tech. These would have to balance the peculiarities of academic
cultures in which these are all formed and cultivated, and hopefully
internalized, and those of the wage labor markets into which students are to be
projected, and the sensibilities, cultures and expectations of employers,
judges, and clients. All of this will call for hard choices and will deeply affect
faculty, perhaps requiring some sort of "making one's peace" with
whatever it is that is emerging in these respects.
b. Faculty. Again
this can be subdivided into two principal categories as follows:
The first is internal — training faculty in tech based and
AI systems and their use in law and legal settings. Even if faculty determine
that they want nothing to do with tech and tech based lawyering, it is
important for them to understand the tech based environment that will be
growing around them.
The second is external — the focus here is on the nuts and
bolts of embedding AI and other tech based mechanisms into the curriculum
generally and in specific doctrinal and clinical courses specifically. It also
involves what may become a quite interesting conversation about the future of
legal writing and the role of library faculty in the context of tech based and
AI mechanisms to the extent they become increasingly relevant in lawyering, in
judicial systems, and elsewhere (including client use and expectations of AI
and tech based facility by lawyers). Indeed, my earlier work studying the AI
policies of twelve institutions documents that library faculty at multiple
institutions (USD, Stanford, Penn State, Michigan, UCLA; Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies; SSRN HERE) are already curating
AI policy resources and template language. This suggests that library and legal
writing faculty are already taking a leading role and serving as human
infrastructure for AI integration.
c. Institutions.
Institutions serve as the platforms (to use a contemporary signification of
what are ancient collectives) where knowledge is both produced and consumed,
and within which the guardrails of that production and consumption is made
authentic, legitimate and projectable outside of the platform itself (into
labor markets for students and prestige and reputation markets for faculty, and
hierarchy setting processes for institutions). Institutions provide the meta
rules within which students and faculty operate and also the rules within which
the institution itself chooses to exist — in this case with respect to tech,
tech based and AI processes and systems. While it has been fashionable to
produce institutional rules from the top down, even when softened through what
appears to be bottom up driven engagement (in liberal democratic style which in
some respects appears to be more akin to Chinese Leninist Mass Line
techniques), this is one case where the institution would do well to wait and
absorb the learning of bottom up efforts; efforts that they might or ought to
be inclined to encourage. Still, institutions are also captives of their own
ruling circles and the expectations that those circles embrace may require the
production of some sort of "action" and especially one that
replicates the movements of university "herd" (politely understood as
the product of benchmarking). Beyond that unavoidable action well outside of
the control, or influence of faculties and colleges, perhaps the better path at
the moment is to encourage college level institutions to begin to develop a
sense of practices and needs that are field specific and that may, over the
next several years draw on the experience of faculty on the ground and
experiments at college level policy. All of this, of course, can be guided and
managed from the top to suit their own agendas and needs (as a function of
trustee demands, the political environment, university level politics, etc.).
All of this is interesting, of course, but of little help to
those university productive forces on the ground — students, front line
administrators and faculty. For them, institutions serve a critical role,
though one that in our culture appears to be undertaken in what to outsiders
(like me) appear to be functionally passive: active coordination and multi-vector engagement, not top down but side
to side. While the university is, like many other institutions in other
economic and social fields, quite happy to proliferate any number of
institutional organs dedicated to some engagement with AI and tech and to
produce resources that advance their "missions" in that respect, the
university, and especially its leading organs, tend to give shorter shrift to
the critically important tasks of (1) coordinating these efforts; (2) fostering
robust communication within the university community (the creation of a university
website, ritual performances of publicity at scattered events, etc. are hardly
enough except as window dressing).
d. Tech. Tech
issues may themselves be divided into at least three sub-categories:
First, AI or tech based systems in class. That was the focus
of my model AI in Class Policy template. Its focus is solely on work produced
for evaluation and work and leaves untouched other use of tech around those
work products which are its subjects. Yet even this is problematic in a general way (see On the Nature of Human-Machine System Interaction: A Conversation with Claude, Harvey AI, Gemini, ChatGPT and Grok (quoting ChatGPT: "Where does "the machine" end? At: the model weights? the inference process? the serving infrastructure? the data centers? the training corpus? the developers?
the optimization process? the users? the network of institutions maintaining the system? There is no purely self-evident answer."))
Second, AI use generally by students beyond the narrow
category of assessment generating work. These then touch on the larger issues
of AI and tech proficiency, ethical use, cultural expectations, practice
cultures and the like. They are also directly linked to the use of AI and other
tech by students in class and otherwise as study aids and the like. My essay on
human-machine interaction documents that AI systems are "structurally
disposed to tell you what you want to hear" and are "structurally
incapable of producing reliable friction." This finding has direct,
consequential implications for students using AI in legal education — it means
that when students use AI for "brainstorming topics or organizational
structures" or "identifying counterarguments" (uses permitted
under my policy template), AI will tend to validate rather than genuinely
challenge their analysis. The stress test report would be substantially
strengthened by integrating this insight: the sycophancy problem is not just a
philosophical curiosity but a pedagogical
problem that the policy template must account for. How does one teach students
to extract genuine analytical value from a system structurally optimized to
agree with them?
e. Tech and AI systems
as stakeholders. This is a hard one for humans, but it is worth considering
whether any tech or AI-focused policy is complete without considering the
fundamental character and responsiveness of computational systems themselves. My human-machine interaction essay (Rethinking AI Governance in Legal Education -- Five Machines (Grok, Harvey, ChatGPT, Claude, and Gemini), One Question, No Consensus but Five Archetypes; SSRN HERE) suggests the way that
different AI systems (Harvey AI, Claude, ChatGPT, Gemini, Grok) have
meaningfully different "floors" and structural dispositions. But the
stress test treats "AI" as a monolith. Which systems are
students actually using? Does it matter that Harvey AI, as a legal-domain
platform with an institutional partnership at Penn State, operates differently
from general-purpose tools? In my class and orally I tried to stress the difference
in operating system "personality" among AI machine systems. But much
more may be necessary. At a minimum, it may be useful to more strongly
foreground that AI in legal education should at least flag that "AI"
is not one thing.
Additionally, my essay on human-machine interaction also suggests that AI systems are
"structurally disposed to tell you what you want to hear" and are
"structurally incapable of producing reliable friction." This finding
might be read as having direct, consequential implications for students using
AI in legal education — it means that when students use AI for
"brainstorming topics or organizational structures" or
"identifying counterarguments" (uses permitted under my policy
template), the AI machine systems utilized will tend to validate rather than
genuinely challenge their analysis. The stress test would be
substantially strengthened by integrating this insight: the sycophancy problem
is not just a philosophical curiosity but a pedagogical
problem that the policy template must account for. How does one teach students
to extract genuine analytical value from a system structurally optimized to
agree with them? That pedagogical problem is augmented when one understands that from a machine-computational perspective, none of this matters; that while humans struggle to manifest communication in flattened sequential block chain type nodes of text that are linear and time dependent (thus the critical element of sequencing in human communication), machine systems are always in the present. They produce an instantaneous picture of a flow that is not temporally contained and then struggles to translate and reduce its computational analytics to the flattened and linear communication of humans. From a machine perspective human efforts at AI policy are just another set of bits that may hold meaning for humans, and the machines would "understand" that meaning relationship in terms of patterned analytics, but which are otherwise meaningless (in the sense of holding no meaning) for machine computational systems (Structure, Legitimacy, and the Limits of Machine-Centered Derivation: An Analysis of Five AI Systems' Third-Stage Attempts to Construct Machine-Centric Governance Policies for Legal Education; SSRN here).
* * * *
Well, there it is, a rough and preliminary assessment of a
stress test conducted in my classes, some very preliminary insights one might
be able to draw from them, and the consequences for governance at every level
of the university, but with particular emphasis on the triadic relationship
between students-instructors-law practice. As expected, the stress test
revealed the significant gaps in knowledge that require further examination. At
the same time it reveals a surprisingly strong and resilient student body that,
even under conditions of high risk and stress, are willing to meet the
challenges under primitive and underdeveloped conditions. They are the
"stars" of this exercise and a continuing source of inspiration. At
the same time, the surfeit of knowledge about what is going on in key wage
labor markets and among other key stakeholders in the production of legal
products and services requires substantial correction. Hardest of all, perhaps,
will be the consequential effects on faculty and university institutions. It
may no longer be crystal clear that a purely reactionary position — the
"just say no" approach — to tech provides a long term solution, as
comforting as that position might be to traditionalists. Things are changing,
not just tech but also expectations and understanding about the role of humans
in the production, dissemination and management of knowledge. Public
organizations (certainly outside the US but here as well), law firms,
judiciaries, and clients are all grappling with these issues. If law faculties
are to remain relevant perhaps we ought to as well.
NOTE: Harvey AI was used to review the text, correct typos, and make comments, some of which I incorporated into the revised text
 |
| poster created with ChatGPT |
