Automated Law--Questions and Answers on Data Driven AI Enhanced Governance and the Emerging Instruments of Social Control

(Pix © Larry Catá Backer 2020; Utrecht Museum Speelklok)

Ratings based data driven frameworks for managing human and institutional behavior are increasingly acquiring an autonomy from the legal and accountability systems they once were thought to serve, and the norms of which were presumed to constrain their utilization. As both the scope and locus of law and legal orders have changed, so has its expression as "text" generated through a legitimating structure tied to the rationalization of political power through public institutions tied to principles of representative capacity and principles based constraints. In simpler terms--the once self evident relationship between law (the command and the normative framework for managing behavior) and its implementation (through measurable accountability measures) has broken down.

One can no longer approach law with the simple certainties inculcated in social actors a generation ago.  Of course, law,. traditionally understood, continues to serve as the most successful product of states for mandatory internal consumption by those within its jurisdiction. Globalization has also transformed law into a commodity, a factor in the production of wealth through economic activity that is proffered by states to economic enterprises for their consumption in global markets for law. Moreover, the pattern of delegating governance authority by states to private organs has accelerated in the face of globalization and in the shadow of the rise of compliance and accountability cultures as the basis for regulation (its ecology is discussed here). Both factors--compliance/accountability cultures, and delegation of the obligation to administer systems of the implement public law based objectives--have moved governance from the simple (though burdensomely overwrought) apparatus of the 20th century administrative state, to the multi-leveled and polycentric systems of data driven accountability founded on observations to meet certain normative expectations but administered by (increasingly non-state) entities so tasked.  

This is fertile ground for the development of a new form of law making, that is for the emergence of automated law. This form seamlessly fuses the normative objective with consequences built around measurable success in meeting those objectives. In lieu of law, administrators and courts, one has normative objectives that can be reduced to measurable goals, against which behavior can be measured, can be rated, through the application of data driven analytics.  Those analytics reduce data to an assessment, usually a number or other value identifier. That assessment, in turn, is fundamentally relational--that is the nature of ratings.  The rating number usually suggests a relationship between what has been measured  and the objectives or ideals against which it has been measured (e.g., a position on a scale). But more importantly, the number can be compared to the numbers produced by the assessment of other persons or institutions in the same ratings class (e.g., all four year U.S. universities, and the like). That number, alone or in combination with other ratings, can then also give rise to consequences--privileges or penalties depending on the application of consequence imposing algorithms.  The process, increasingly as been data by the technologies of big data management, and made more efficient by machine learning programs to which these consequence bearing decisions may be allocated.  

For traditional law and its protectors--these trends pose multiple challenges. And yet those challenges may not be resolved merely by wishing away either compliance and accountability cultures, the power of consequence bearing analytics, or the power of ratings as a micro-enforcer of norms beyond courts.  The result produces not just strong pressure on law to change its relationship to politics and the administration of power, but also to the transposition of the constraints that have contributed to the legitimization of law systems onto these new modalities of regulation. Automated law bears watching; it imposes on the human community for whom it was created, to ensure its integrity and conformity with those values that enhance societal well being (understood as cultural-political choices, and thus subject to the vagaries of politics and its own regulatory and integrity challenges).

I was recently given an opportunity to think briefly about some of the questions raised by these challenges in an informal interview, the gist of my answers to which follow. The final version of the interview will form part of a group of contributions built around the Superscoring Symposium sponsored by the Grimme-Institute (Germany) and funded and supported by Grimme-Forschungskolleg an der Universität zu Köln and Bundeszentrale für politische Bildung (bpb). A draft of my forthcoming article, Automated Law: Blacklists and Social Credit Regimes in China.



Interviewer: We take “super-scoring” to mean any accumulation and combination of personal data taken from different contexts and sources and used in an algorithmic production of one functional value that is intended to evaluate human behaviour.

Question 1. What is your evaluation of super-scoring as a tool for social control? 

I believe that the answer to this question depends on one’s time reference. In the short term it is not clear that super scoring is as yet developed well enough to serve as a tool for social control. The tool is still very much in its infancy. It has neither the structures nor the institutional maturity needed to be fully deployed. More importantly, the critical alignments between the private sector, to which the implementation of such super scoring tools will be ceded, and the public sector, whose objectives articulated through law are necessary to develop the framework for seamless application, has yet to be developed. For the moment one observes a period of formation and experimentation, one mostly driven by markets and the private sector. In the long term, however, super scoring will likely be a quite powerful tool for social control. The issue then will be for whose behalf will such social control tools be utilized. On the one hand the tool is most legitimately deployed by the state. However, it is not clear that the state ought ot have the authority to engage in such acts of social control--especially where the result involves social engineering that effectively changes persons or a people, at least without their consent. The issue becomes more complicated in the West where the state has shown a propensity to delegate its use and deployment to the private sector. If there may, or ought to be, questions about the extent to which a state, consistent with its national traditions and expectations, those questions might apply with equal strength where such power is deployed by private actors. Yet private actors have always sought to engage in shaping culture, and social expectations. Ratings based governance, however, no longer appear consigned to lower order administrative management, or to the cultivation of consumer tastes. Rather, they are now powerful enough tools to substitute fro law as a method of changing the baselines against which culture is understood and practiced.  It represents the possibility of a quite precise rationalization of human management in the way that scientific approaches have brought a measure of proactive interventions to the husbandry of farm animals.  Whoever controls superscoring systems will tend to control the narrative that moves the social order forward. And lastly, of course, the day will likely come when superscoring may well be a quite useful tool not for social control but for the assessment and accountability of the state itself.

 Question 2. Where and for whom do you expect to see the greatest opportunities and risks? 

The greatest immediate opportunities for superscoring systems are in the private sector. More specifically, and arising from long practice, the financial services sector. In addition, those enterprises capable of harvesting, packaging, and selling data are likely to profit from the move toward data driven governance through ratings and super scoring,. The challenge is the traditional one for a new industry—the rationalization of production to offer products that meet consumer demand. The greatest risks, of course, are those of the objects of superscoring—those who will be subject to regimes of rewards and punishments on the basis of their score. Data integrity, data verification, corruption, defective analytics, and fairness in the construction of analytics and algorithm will pose the greatest risks.

Question 3. Do you anticipate upcoming conflicts between super-scoring systems and established values and norms? 

Those conflicts are already in the offing. The harmonization of accountability and compliance based scoring systems with the traditional cultures of rechtsstaat will require a rethinking of the hierarchy of law, and its characterization as purely an act of state organs acting under authority of higher law. More specifically, the difficulty involves the harmonization of the cultures and habits of a law based system of behavior control—centered on the exercise of the prosecutorial and judicial authority of the state—with that of an administrative and technical based system—centered on the exercise of administrative discretion by officials and routinized compliance based assessment.At the same time, law and scoring systems have managed to avoid confrontation for a number of important reasons.  First, the traditional hierarchical connection between law and scoring continues to have some viability.  That relationship consigns to law the normative  task, and to ratings systems the "engineering" or implementation function.  But that relationship becomes more illusion that fact where the scoring itself fills in gaps left by legal systems that produce either general principles-objectives or which are focused on the provision of the legal basis for (of the regulation of) super scoring institutions.  Second, those in a position to consider the role of superscoring have tended to underestimate both  its significance and the extent of its use. As a consequence,  normative considerations of regulatory systems grounded in scoring were viewed as irrelevant to a perceived second order system that was fit only for technicians. The view in Marxist Leninist states, particularly China was different, of course.  There super scoring has been both a public  undertaking and a moral one in line with that political-economic model. Third, the normative implications of law and rule systems continues to be relegated to politicians or philosophers. The moral element of jurisprudence has given way to politics, and politics now is indifferent to method.  Fourth, to the extent that normative legalism is relevant at all, it tends to appear in the service of the protection of the domain of law as a field (and its traditional elements--statutes, administrators, courts and the dominance of the public sphere and it regimes of constitutional norms). It os not clear that the structures of analysis and the parameters of control developed for law systems have a substantial overlap with issues of control and deployment of ratings systems and their AI enhanced algorithmic frameworks of consequences. In that respect much work remains to be done.  At any rate the current approach of developing principles of ratings enhanced AI that effectively extend the constraints of rule of law (developed for statutes and administrative operations) may not serve as intended.  

Question 4. How do you expect super-scoring processes to shift with the implementation of certain digital technologies (such as smartphone tracking, face and voice recognition etc.)? 

Super scoring processes will be dependent on the efficiency of data harvesting technologies, to be sure. Yet one ought not to necessary equate the power to generate gross amounts data with the ability of organize and use that data to specific ends, and to use it well. The technologies of data harvesting will certainly shift power, in markets, toward data gatherers, at least in the first instance. But rich markets invite competition. And data is merely the start of a complex process of production, which requires the application of analytics and the wise development of algorithms. As such, it is not the process but the power of super scoring systems that will shift with the advance of technologies that can more efficiently produce data, but data useful to feed the analytics that are the foundation of the ratings at the heart of super scoring. However, what does change, and what really serves as the great break with traditional structures of law, is that this "next generation" legal system is automated.  That is, the distinctive feature of superscoring systems, whether in its Western privatized forms or in the current efforts to construct a coherent national Social Credit System in China, is that law requires less human intervention in its construction and operation.  Automated law works autonomously--it is for that reason, for example, that one worries about AI and machine learning systems.  Where law is reduced to the development of measurable objectives against which human behavior is assessed, and thus assessed either rewarded or punished (with the scaling of rewards and punishments also rationalized), then it is possible for machines to apply the systen without huma intervention,.  And indeed, from a certain perspective, automated law solves one of the great challenges of the 20th century administrative state (whether overseen by a Marxist Leninist Nomenklatra or by a Western corps of professional administrators), the problem of corruption.  Of course having contributed to the amelioration of one problem, technology might well contribute to a host of others: systemic corruption, data integrity; imperfect analytics, and misused algorithms. Those are the areas in which law might find a new space for application.  But that space requires that law be willing to discard its historical conceits.

Question 5. What kind of interplay can we expect between the digital economy and politics if super-scoring systems are implemented -- on a regional, national or international level? 

One ought to expect both an alignment of objectives, and purposes to some extent, even as the delegation at the heart of super scoring systems will likely also further the autonomy of privatized entities central to the production of such ratings. Of course rating systems will likely be divided between public and private function scoring. The state will turn to its administrative apparatus to administer a super scoring system to more efficiently manage the technical requirements of its administrative regulatory apparatus. The private sector will continue to rely in super scoring to its own ends, each according to the logic of the industrial sector in which it operates. And of course, the management of markets for data will present one of a number of areas in which the state and private actors will likely engage in dialogue—the state to ensure that such markets do not otherwise violate public policy or law; the private actors to ensure that market efficiencies will produce welfare optimizing results. But that is precisely what private enterprises and the state have been engaging in sometimes heated political debates for over a century. Thus perhaps it is best to express what will emerge is old wine in new bottles. More worrisome will be the governance effects of super scoring when undertaken by and for the interests of private actors. In those cases one worries that politics and power will shift from the state to the private sector. For the state, this presents a regulatory challenge that aligns with its markets ideologies. For the private sector it presents an opportunity to control the social narrative and thus to manage the political landscape within which it operates.

Question 6. Which aspects of super-scoring should concrete educational measures address? Where and how would you begin to do so? 

The educational aspects of super scoring presents the academy with a tempting trap. The easiest response of academic institutions to the emerging realities of data driven governance would be to focus on the development of the technical skills necessary to facilitate the tasks of data harvesting, analytics and algorithmic construction. Among the most abstract and important of the educational objectives would be not just the blue-collar (technical) work (such as coding, elementary analytics, and the like) but the science of machine learning, big data management and artificial intelligence. And there lies the trap. These skills standing alone ignore the moral element of super scoring. It ignores the teaching of judgment necessary to align the technical construction of superscoring systems with the protection of the moral values on which the political society is founded. Failure to teach such moral skills and to align them with the teaching of the more technical aspects will produce the sort of amoral systems that will detach superscoring from the political-ethical foundations of the social-moral order.

Question 7. What aspects of super-scoring do you believe to be under-represented in current public discourse? What other questions should be raised? 

The most important lacunae of current public discourse touch on issue of coherence across policy areas, as well as the stubborn determination by lawyers and others to impose 20th century mechanics to the solution of the sui generis challenges of a regulatory system not grounded in the modalities of the traditional law-state. Society has moved far from the issues that confronted the state in the aftermath of the violence of the first half of the 20th century. We no longer face the abyss of barbarity perpetrated in the name of the state—at least among the large community of nations. Rather we face the relentless cold and extra-moral cultures of an administrative apparatus that operates on the basis of the exercise of discretion and the routinization of behaviors. Those tendencies pose the greatest risks of dehumanization in the first half of this century and in the form of super scoring can erode the core notions of human dignity in ways that neither law nor morals is yet prepared to meet.