 |
| Pix credit here |
I have been stress testing an application of a model AI Classroom policy that I developed over the summer 2026 (see "AI assists. You think. You analyze. You write. You take responsibility": Creating a Course AI Use Policy Template --Policy Text, Justification and Rule Summary for My Law & Religion Class at Penn State Dickinson). The test subjects were (1) the students in my class, one a large and traditional advanced basic course in corporations, the other a small class on advanced issues in the Constitutional law of religion; (2) me, the faculty member who is meant to model, apply, oversee, and ensure positive outcomes, even under conditions of learning "struggle"; (3) the institutional capacity to support and absorb stress, failure, success and to be nimble in the face of changing conditions. It is a stress test loosely in the traditional sense — a form of deliberately intense or thorough testing, used to determine the stability of a given system, critical infrastructure or entity — though hardly a proper scientific test at this stage, where the object is to figure out the terrain and the points where more finely tuned and proper scientific study may be attempted. The experience is ongoing but a number of insights begin to take form:
1. Students, collectively, may not be ready. I overestimated student knowledge of, experience with, and readiness to attempt to integrate AI and tech based products into their work. And of course, the range of student experience with, desire to use, and understanding of tech in general and AI in particular, as applied in a classroom setting is profoundly varied. Even if they have experience, they have no working knowledge, for the most part, in translating general experiences with AI and other tech based modalities into the specifics of AI and other tech based use in legal work, and more specifically, in legal work within an educational institution the quirks of which may be different than those of legal practice, government, enterprise or other work. A substantial amount of empirical work is required to get a better sense of the state of student knowledge and experience to better refine the embedding of tech and AI based knowledge production and dissemination into a classroom setting. And even more work is required to understand how that process is being evolved with courts, administrative agencies, and law firms. The first step might include developing polling data on current students and incoming students to get a better sense of institution-specific characteristics of student populations, from which policy and action may be better aligned with realities.
2. Students are remarkably resilient. For all of that, the first several weeks of class — the most stressful part of this experiment on student capacity — went surprisingly well. It went remarkably well despite any number of glitches on my part, and the need for students to develop new talents and adjust to new forms of knowledge dissemination and production for which they had little experience to draw on from their years of schooling. It is true enough that students struggled, but they struggled in a positive way (even if they might not have thought so). A few preliminary observations, though, are worth making:
a. Front loading actual AI work in the beginning of the semester is not a good idea. It was the product of my overestimation of student experience with and comfort around AI and other tech based systems. That was a mistake. A better approach would be to first produce a series of training exercises, in the manner of the training for BOTS, and then to use that as the base for the blending of tech and normative learning. That is a project I am now starting to undertake, but I suspect it will produce its own challenges. There is another reason as well. I found, at least this term, that students are risk averse. This is not a criticism: within learning cultures in which everything is assessed and in which assessments are critical to sorting processes for student insertion in wage labor markets for which educational institutions play a part, students will tend to assess the value of a course at least in part as a function of the risk of adverse assessment, which is in turn assessed as a function of their assessment of their capabilities and the time and interest they might have given a particular set of risks. I have no quarrel with that. However, given the realization of the amount and character of work now expected of them (moving from the traditional passive to a far more active role in learning) and the need for assessment as a function of the add/drop period, I should have assumed a substantial probability of substantial movement in but mostly out of a course the risk of which was hard to assess but likely higher than traditional courses. The resulting movements caused substantial problems with group assignments and combined with front loaded group work augmented challenges for students and for the instructor as well.
b. Student learning cultures must be reframed with sensitivity to the depth and naturalization of inculcated educational processes and expectations which are now naturalized within student cognitive "expectations" of a "learning" and "classroom" experience. That requires a broader conversation which will be difficult and ultimately a slow and likely reluctant change in such expectations from faculty and educational institutions. That is likely to take at least a decade given the cultural conservatism of institutional sectors, and the current system grounded in a sort of computational sycophancy system in which processes are produced as a mutual accommodation of student desires and educational path dependent processes.
c. Students prefer substantial guidance, especially at the beginning of a course. Part of the challenges that emerged with the course design was the risk I took in combining two discrete changes. The first was of course the introduction of a heavy tech component. The second was a more robust move toward student driven learning, something that I had been working toward slowly over the last decade. This year I added greater student learning autonomy guided through the production of very detailed student teaching notes that I prepared in lieu of a lecture and that covered virtually all of the contents of a class lecture drawn from the materials, and a set of Infographics that provided a visual aid to the learning to be undertaken for each class. That appears not to have been sufficient; at least for the first third of the term. My sense is that even if the instructor adds little more to the materials, their presence appears to be a critical performative element of student learning, at least in the initial part of a semester.
With this in mind I will now try an alternative and consider its value both for pedagogy and effective student engagement. I will stretch each group assignment into two classes. For the first of the classes student groups will meet and work on their presentations, including the background papers and PowerPoint, During class my TA or I will walk around , observe, and help. Student groups will then make their oral presentations on the following day. The down side is that it consumes a bit of class time. The benefit, to be tested, is that it will add an incrementally significant element to learning,
d. Grading can be an impediment to learning and certainly to risk taking. By this I do not mean course grading; I mean the assessment of the activities of students now fashionably broken down into smaller bits each of which serves as an evaluation layer that have become a central focus of student strategic behaviors. Not that I blame them; they are rational actors; and not that faculty were foolish, the intent was to enhance learning and reduce the stress of a single assessment. My mistake was to grade the exercises under conditions of heightened risk (especially of the unknown) — that is of risk with respect to which prior student experience offered little by way of assessing risk. I will now no longer grade the initial exercises — they are a learning experience rather than an assessment exercise. I will evaluate them as a learning tool.
The deeper question — what constitutes valid assessment when AI is embedded in the production process — is not addressed. My policy template's certification requirement "converts the disclosure obligations into an affirmative representation," but the stress test does not discuss whether students found the certification process intelligible, whether the appendix requirements were practicable, or whether the 100-word / 10% / 30% numerical ceilings functioned as intended in practice. This is something I will consider further with students.
e. Student equity issues. The policy template developed for the courses (see "AI assists. You think. You analyze. You write. You take responsibility": Creating a Course AI Use Policy Template --Policy Text, Justification and Rule Summary for My Law & Religion Class at Penn State Dickinson) explicitly flags equity concerns — "students with more experience prompting AI tools, or with access to premium tools, could gain a differential advantage" — and frames disclosure as "a necessary first step" in exposing that differential. The stress test appears to confirm a "profoundly varied" range of student experience. It may be necessary to consider a more robust approach to these disparities. Two issues merit further study: whether equity issues ought to affect consideration of the role of evaluation rather than grading at least in the initial projects, and whether there is a way to more precisely measure differentials other than qualitatively. That discussion, in turn, may be a function of the goal of embedding AI in coursework, and in that connection whether this suggests a stronger case for developing an AI and tech based skills/practice curriculum.
3. Faculty, collectively, may not be ready ready. Let me raise several general classes of issues preliminarily:
a. Realistic faculty baselines. I underestimated the role of faculty in guiding AI use and miscalculated the guidance necessary to embed AI in student work. More interesting, for me, was a failure to note that the connection between technology and knowledge production would be readily apparent and easy to manifest in faculty and student work. I also underestimated the amount of learning that is required to fully develop an integrated set of teaching pedagogy that integrates AI learning with the traditional flows and substantive knowledge dissemination expectations of students. Moreover, faculty mirror students in both experience with AI, and a willingness or interest in using AI.
b. The effects of tech and AI on not what we teach but how we understand the objects of instruction. As important, faculty like many law firms with whose members I have spoken, are at once aware that technology changes the relationship of the human to knowledge generation and production, as well as knowledge dissemination, but at the same time no one of my acquaintance has yet to claim a firm grasp of the way that tech, and especially AI tech, actually changes the relationship between what it means to produce knowledge, to curate knowledge production, to deploy knowledge nor the way that human machine interactions changes the relationship between knowledge production and the role of humans in it. Many of the people of my acquaintance hold fast to the ideals of the pre-tech age in those regards. It is not clear that those expectations, now pretensions, will be relevant much longer. But the alternatives are hard to describe much less to analyze in a useful way.
c. Consequences for any "Model AI in Class Policy" template. My Model Policy Template is built around a "narrow, conditional exception layered onto a default of independent work, with permission calibrated to where oversight and iterative disclosure are feasible." The stress test suggests that students may lack the baseline AI literacy to meaningfully comply with disclosure and appendix requirements (Section 1), and that front-loading AI work was premature (Section 2a). It may follow that it is necessary to circle back to ask: what does this mean for the policy template itself? Does the template need a phased implementation protocol? Should the disclosure-and-appendix architecture be preceded by a training module? My template acknowledges it is "designed to be tested with students in an actual course setting and refined based on what that experience reveals," but the stress test also suggests that further refinement may be required to make the policy more relevant or at least connected to student realities, at least as they exist now.
4. Institutions, as a collective, may not have built capacity to be ready. That produces an institutional issue. Might it be necessary to develop initial evaluation of student knowledge of AI systems and use; if that is the case does that suggest the need for separate AI specific training, perhaps as part of a first-year curriculum and as an augmentation of the traditional courses in legal writing — now legal writing and tech? These are questions we have not yet even begun to ask. At a higher level of institutional ordering one might also have to ask about differentiation between U.S. and non-U.S. students in skill sets and more importantly in needs and expectations in home countries. Indeed, this is no small matter, and a complicated one. It might be necessary to develop at least some sensitivity to the issue, and it also might suggest that AI skills courses would have to be developed with this fundamental differentiation in mind. On the other hand foreign students seeking grounding in U.S. approaches might be most well served by deep integration into U.S. approaches and expectations in the utilization of AI tech. The same set of considerations may apply to faculty who might require evaluation and training. But here the institutional environment becomes more complicated because one has yet to confront the issue of AI use as a function of academic freedom.
5. Education cultures make the introduction of AI based teaching and knowledge dissemination/production more difficult. My prior work suggests that while the general population of stakeholders in law schools may be progressive in matters of politics and culture, they appear to be quite conservative and traditionalist, one might be tempted to say reactionary, in the face of the potentially sweeping changes that technology, and especially AI based tech, may bring not just to the practices of education but also to its fundamental pedagogies and working styles. The current trend, marked by wariness and a muscular effort to preserve the present against the possibly transformative (or corrupting) potential that tech in general and AI in particular brings, is likely to remain dominant in the short term and then necessarily change as judicial and legal or attorney practices (and expectations) change. But that requires ongoing faculty discussion and before that perhaps some "bringing up to speed" exercises (and also for students). These in turn will clarify and more precisely describe the choices going forward.
6. AI adds another level of stress for teaching and learning that may produce both student and faculty resistance. AI adds a layer of uncertainty — both in terms of substantive knowledge and in process expectations — that increases risk, and thus student stress, in an environment in which every response is both a test and an evaluation. In evaluation orderings, like law school, where every assessment can contribute to micro movements in relative performance status which affects student understanding of successful insertion in legal labor markets, the addition of a great unknown, AI, as a substantive field of learning and as a process embedded in learning a traditional field of knowledge, can add stress. That stress has two parts — the first is in the mastery of the subject, the second is in the ability to properly apply it under conditions of substantial uncertainty.
7. It is not yet clear what is the nature and role and situating of AI and other tech based innovation with effects on legal practice. That is particularly, and perhaps acutely, the case with respect to AI in relation to fields of knowledge and student/faculty roles. Legal tech could be described in good faith as its own sub field. That, in turn, would suggest that legal tech might be best suited, at least in the middle term, as its own course, as a separable object of study. Legal tech, on the other hand, might be understood as instruments available for legal study and in that sense ought to be part of the pedagogy of normative courses and clinicals, each modified to suit the field of study. It is likely more reasonable to apply the legal writing analogy to tech — that is it is both its own field of study and deeply embedded in substantive and clinical courses. That may, in turn, require substantial curricular changes.
8. Disaggregating Issues. As an initial effort to disaggregate the issues around tech and tech based education I have attempted an initial categorization of issues as follows:
a. Students. This can be subdivided into three sub categories.
The first is external — managing the environment in which students may use, and learn to use, tech based and AI mechanisms in the production, dissemination and analysis of legal issues relevant to their courses and ultimately to their work as lawyers. To that effect the traditional focus on classroom model AI policies and more generally law school or institutional guardrails for use of AI and other tech by students is a central element of contemporary management.
The second is internal — training students in the use of tech based and specifically AI grounded "tools" (the word is in quotes precisely because it is not clear that AI is merely a passive tool. Nor is it clear that AI will be treated merely as an instrument for much longer).
The two are intimately connected but not yet in ways that it is clear we understand well enough to act, even in the short term. Still the dialectical relationship provides a basis for developing plans and modalities for advancing student learning and student learning environment well enough.
The third is the cultivation and management of student training, expectations, risk assessment, and ultimately of their baseline engagement with tech. These would have to balance the peculiarities of academic cultures in which these are all formed and cultivated, and hopefully internalized, and those of the wage labor markets into which students are to be projected, and the sensibilities, cultures and expectations of employers, judges, and clients. All of this will call for hard choices and will deeply affect faculty, perhaps requiring some sort of "making one's peace" with whatever it is that is emerging in these respects.
b. Faculty. Again this can be subdivided into two principal categories as follows:
The first is internal — training faculty in tech based and AI systems and their use in law and legal settings. Even if faculty determine that they want nothing to do with tech and tech based lawyering, it is important for them to understand the tech based environment that will be growing around them.
The second is external — the focus here is on the nuts and bolts of embedding AI and other tech based mechanisms into the curriculum generally and in specific doctrinal and clinical courses specifically. It also involves what may become a quite interesting conversation about the future of legal writing and the role of library faculty in the context of tech based and AI mechanisms to the extent they become increasingly relevant in lawyering, in judicial systems, and elsewhere (including client use and expectations of AI and tech based facility by lawyers). Indeed, my earlier work studying the AI policies of twelve institutions documents that library faculty at multiple institutions (USD, Stanford, Penn State, Michigan, UCLA; Structure, Opacity, and Convergence: A Consolidated Analysis of Law School Generative AI Coursework and Exam Policies; SSRN HERE) are already curating AI policy resources and template language. This suggests that library and legal writing faculty are already taking a leading role and serving as human infrastructure for AI integration.
c. Institutions. Institutions serve as the platforms (to use a contemporary signification of what are ancient collectives) where knowledge is both produced and consumed, and within which the guardrails of that production and consumption is made authentic, legitimate and projectable outside of the platform itself (into labor markets for students and prestige and reputation markets for faculty, and hierarchy setting processes for institutions). Institutions provide the meta rules within which students and faculty operate and also the rules within which the institution itself chooses to exist — in this case with respect to tech, tech based and AI processes and systems. While it has been fashionable to produce institutional rules from the top down, even when softened through what appears to be bottom up driven engagement (in liberal democratic style which in some respects appears to be more akin to Chinese Leninist Mass Line techniques), this is one case where the institution would do well to wait and absorb the learning of bottom up efforts; efforts that they might or ought to be inclined to encourage. Still, institutions are also captives of their own ruling circles and the expectations that those circles embrace may require the production of some sort of "action" and especially one that replicates the movements of university "herd" (politely understood as the product of benchmarking). Beyond that unavoidable action well outside of the control, or influence of faculties and colleges, perhaps the better path at the moment is to encourage college level institutions to begin to develop a sense of practices and needs that are field specific and that may, over the next several years draw on the experience of faculty on the ground and experiments at college level policy. All of this, of course, can be guided and managed from the top to suit their own agendas and needs (as a function of trustee demands, the political environment, university level politics, etc.).
All of this is interesting, of course, but of little help to those university productive forces on the ground — students, front line administrators and faculty. For them, institutions serve a critical role, though one that in our culture appears to be undertaken in what to outsiders (like me) appear to be functionally passive: active coordination and multi-vector engagement, not top down but side to side. While the university is, like many other institutions in other economic and social fields, quite happy to proliferate any number of institutional organs dedicated to some engagement with AI and tech and to produce resources that advance their "missions" in that respect, the university, and especially its leading organs, tend to give shorter shrift to the critically important tasks of (1) coordinating these efforts; (2) fostering robust communication within the university community (the creation of a university website, ritual performances of publicity at scattered events, etc. are hardly enough except as window dressing).
d. Tech. Tech issues may themselves be divided into at least three sub-categories:
First, AI or tech based systems in class. That was the focus
of my model AI in Class Policy template. Its focus is solely on work produced
for evaluation and work and leaves untouched other use of tech around those
work products which are its subjects. Yet even this is problematic in a general way (see On the Nature of Human-Machine System Interaction: A Conversation with Claude, Harvey AI, Gemini, ChatGPT and Grok (quoting ChatGPT: "Where does "the machine" end? At: the model weights? the inference process? the serving infrastructure? the data centers? the training corpus? the developers?
the optimization process? the users? the network of institutions maintaining the system? There is no purely self-evident answer."))
Second, AI use generally by students beyond the narrow category of assessment generating work. These then touch on the larger issues of AI and tech proficiency, ethical use, cultural expectations, practice cultures and the like. They are also directly linked to the use of AI and other tech by students in class and otherwise as study aids and the like. My essay on human-machine interaction documents that AI systems are "structurally disposed to tell you what you want to hear" and are "structurally incapable of producing reliable friction." This finding has direct, consequential implications for students using AI in legal education — it means that when students use AI for "brainstorming topics or organizational structures" or "identifying counterarguments" (uses permitted under my policy template), AI will tend to validate rather than genuinely challenge their analysis. The stress test report would be substantially strengthened by integrating this insight: the sycophancy problem is not just a philosophical curiosity but a pedagogical problem that the policy template must account for. How does one teach students to extract genuine analytical value from a system structurally optimized to agree with them?
e. Tech and AI systems as stakeholders. This is a hard one for humans, but it is worth considering whether any tech or AI-focused policy is complete without considering the fundamental character and responsiveness of computational systems themselves. My human-machine interaction essay (Rethinking AI Governance in Legal Education -- Five Machines (Grok, Harvey, ChatGPT, Claude, and Gemini), One Question, No Consensus but Five Archetypes; SSRN HERE) suggests the way that different AI systems (Harvey AI, Claude, ChatGPT, Gemini, Grok) have meaningfully different "floors" and structural dispositions. But the stress test treats "AI" as a monolith. Which systems are students actually using? Does it matter that Harvey AI, as a legal-domain platform with an institutional partnership at Penn State, operates differently from general-purpose tools? In my class and orally I tried to stress the difference in operating system "personality" among AI machine systems. But much more may be necessary. At a minimum, it may be useful to more strongly foreground that AI in legal education should at least flag that "AI" is not one thing.
Additionally, my essay on human-machine interaction also suggests that AI systems are "structurally disposed to tell you what you want to hear" and are "structurally incapable of producing reliable friction." This finding might be read as having direct, consequential implications for students using AI in legal education — it means that when students use AI for "brainstorming topics or organizational structures" or "identifying counterarguments" (uses permitted under my policy template), the AI machine systems utilized will tend to validate rather than genuinely challenge their analysis. The stress test would be substantially strengthened by integrating this insight: the sycophancy problem is not just a philosophical curiosity but a pedagogical problem that the policy template must account for. How does one teach students to extract genuine analytical value from a system structurally optimized to agree with them? That pedagogical problem is augmented when one understands that from a machine-computational perspective, none of this matters; that while humans struggle to manifest communication in flattened sequential block chain type nodes of text that are linear and time dependent (thus the critical element of sequencing in human communication), machine systems are always in the present. They produce an instantaneous picture of a flow that is not temporally contained and then struggles to translate and reduce its computational analytics to the flattened and linear communication of humans. From a machine perspective human efforts at AI policy are just another set of bits that may hold meaning for humans, and the machines would "understand" that meaning relationship in terms of patterned analytics, but which are otherwise meaningless (in the sense of holding no meaning) for machine computational systems (Structure, Legitimacy, and the Limits of Machine-Centered Derivation: An Analysis of Five AI Systems' Third-Stage Attempts to Construct Machine-Centric Governance Policies for Legal Education; SSRN here).
* * * *
Well, there it is, a rough and preliminary assessment of a stress test conducted in my classes, some very preliminary insights one might be able to draw from them, and the consequences for governance at every level of the university, but with particular emphasis on the triadic relationship between students-instructors-law practice. As expected, the stress test revealed the significant gaps in knowledge that require further examination. At the same time it reveals a surprisingly strong and resilient student body that, even under conditions of high risk and stress, are willing to meet the challenges under primitive and underdeveloped conditions. They are the "stars" of this exercise and a continuing source of inspiration. At the same time, the surfeit of knowledge about what is going on in key wage labor markets and among other key stakeholders in the production of legal products and services requires substantial correction. Hardest of all, perhaps, will be the consequential effects on faculty and university institutions. It may no longer be crystal clear that a purely reactionary position — the "just say no" approach — to tech provides a long term solution, as comforting as that position might be to traditionalists. Things are changing, not just tech but also expectations and understanding about the role of humans in the production, dissemination and management of knowledge. Public organizations (certainly outside the US but here as well), law firms, judiciaries, and clients are all grappling with these issues. If law faculties are to remain relevant perhaps we ought to as well.
NOTE: Harvey AI was used to review the text, correct typos, and make comments, some of which I incorporated into the revised text
 |
| poster created with ChatGPT |
