Berg's Words

The Greatest Challenge Of Our Time

Software and the people that make it


The greatest challenge of our time

New modes of organizational technology create and sustain empires. Ray Dalio has been discussing this empire thesis for a while and wrote about it most recently here. The Dutch created capitalism and so their empire. The English created mechanized production and kicked off the Industrial Revolution and so their empire. The United States created a system of de-centalized control and so their empire. But technology is contextual and situated. What triumphed in the past may not win now and vice versa. The modes of organization available and their efficacy exist intertwined with the technology and philosophy of the day. We now sit precariously, perhaps on the precipice of eclipse by the East and their centralized authoritarian bureaucracy. For centuries, decentralized decision making, enabled through the miracle of capitalism—the engine that enabled global optimization through locally-optimal choices—, has preserved and bolstered those who embraced it. On the dawn of the big data-driven age of predictive psychohistory, the pendulum seems to have started its traversal towards the side of centralization. Having experienced the horrors of bureaucracy writ small, I cannot imagine that bureaucracy writ large, temporarily ascendant and momentarily more fit than our bastardized capitalistic freedom though perhaps it may be, could possibly be adopted to the long-term benefit of humanity. Centralization and autocratic bureaucracy is an absorbing state whereas decentralized freedom is not. We now face a dual imperative—it is simultaneously our moral duty to work towards the benefit of future humanity and our economic calling to ensure the continued and uninterrupted supremacy of liberal capitalism over authoritarian dictatorship as a means of organization.

The world is a competitive marketplace and the winner wins for real cause. We can argue about might being right but one cannot argue against might being the only means by which to enable right because without might, no perceived optimum is evolutionarily stable. Thus, we must wonder from whence this moral imperative to forestall the rise of authoritarian centralization is derived. Putting aside political philosophy and moral considerations and looking only at efficiency of economic resource allocation (a good proxy for “might”), it would seem that either it is better than our current modes of organization and therefore inevitably ascendant and, to a great degree, actually preferable (there is, after all, little sense in hanging onto the unicorn dreams of a non-equilibrium anachronism), or it is suboptimal and, therefore, non-threatening in any real sense. To this alluring argument I raise two objections. One, in the presence of absorbing barriers, great caution is warranted and, two, the world is at minimum chaotic and, more likely, inherently probabilistic.

Let us imagine a scenario wherein authoritarian centralization is truly optimal for the next hundred years until humanity has developed some set of technologies that enable decentralized superiority once again. But let us also imagine that the core nature of humanity undergoes no wholesale restructuring and personal self-interest continues to motivate individual actors. Finally, we imagine that, as has been the case with all previous such technologies in human history, it takes some amount of time to reap sufficient gains from this yet-to-be-discovered organizational technology as to beat out on an absolute basis the authoritarian centralization ruling the day. Under such reasonable assumptions, even if a small pocket of humanity begins to realize a society organized by this superior mechanism, they will be snuffed out by the ruling central authoritarian regime long before their absolute power can challenge the regime’s and snuffed out with them the hope for human progress; the billions who would come after them made poorer by our inaction now. What action now can prevent this if our core assumption is that centralized authoritarianism is actually optimal over the next hundred years? We must over-invest in bringing about the technological preconditions to support the next optimum now. If we can develop the substrate on which to build the post-centralized optimum before the centralized soon-to-be-optimum eclipses us in absolute power, we have saved the future of humanity by avoiding the looming absorbing barrier, the fixed-point dark age.

Even if freedom remains the best means of achieving human utility and remains a viable equilibrium, it seems incontrovertible that modern technology has narrowed the gap in relative power between a freedom-embracing state, admittedly hobbled though it may be by a persistent drift toward bureaucratic control, and a centralized authoritarian regime. The complexity of humanity and our interrelationships is such that seemingly innocuous stimuli can propagate and constructively interfere in chaotic reverberation throughout the whole of the world. The narrower the gap between best and second best, the less assured the primacy of the best as any slight deviation—a great orator or social media influencer or pandemic or overextended bank or social justice-seeking self-defeating zealots or a lone martyr, murdered at the hands of those supposedly protecting the public—may inadvertently nudge the world toward a new equilibrium, including a second best from which it cannot easily extricate itself. In his Origins of Order, Kaufman talks of the propensity for random variation to swamp the power of natural selection on gently rolling evolutionary fitness hills. By just this mechanism might we face the destruction of lives, by just this means might we impoverish our species, by just this method might we, by constraining humanity to our one world, extinguish humankind from the universe.

Hints of a cause

The giant corporations that politicians lambast for having an ever-growing command of our lives but, as evidenced by their revealed preferences, consumers adore, demonstrate that there are returns to centralization. These companies have chosen to bring multiple services into one firm rather than rely on the market to buy, combine, and resell aspects of the services they offer. Seeing this behavior from all of the most valuable firms in the world evinces a clear conclusion: a basket of intra-firm services generates more value today than a (hypothetical) similar basket of inter-firm services. The centalized authoritarianism of the East is enabled and made ascendant by precisely the same mechanisms as those that enable the Western mega-corporation.

Regulation and morality

First, the moral and philosophical unsuitability of regulation in service of our end.

Intolerance of intolerance places the cart of history on the road to serfdom. If freedom is not an evolutionarily stable optimum then there is nothing of value in it worth protecting and only a philosopher or a doublespeak-trained propagandist could take the claim that we are protecting freedom by constraining freedom as a legitimate thought. If the only means by which we can hope to preserve freedom against the attacking hordes is to occasionally devolve into intolerant authoritarianism, we have created a society well-fit for the hordes. We are right now being reminded how short our remaining leash of freedom is. When you are only free insofar as you stay on the well-marked and well-trod paths of normality, you are most assuredly enslaved. Further, Machiavelli could have told us 500 years ago that the population would better tolerate an authoritarian granting some freedoms over time than a free people being ever more obviously enslaved.

Now, the practical failure of regulation to serve our end.

Regulation misses the point in a competitive global marketplace: if bigger is better, we cannot regulate away bigness—we must invent the technology to beat it or be beat by it. Regulating it in the West only serves to ensure that it will grow unchecked outside the bounds of our regulators’ purview.

That is: if Amazon finds it more efficient to directly employ nearly a million people than to contract between firms, with, for example, warehousing and software development bid out to the most efficient provider, we have less reason to believe that, at least at similar scales, we should obviously prefer market to hierarchical or authoritarian economic organization, purely on grounds of efficiency of allocation of resources. Put differently, there is presently within the economy of the West some evidence at the level of individual enormous firms to suggest that the Eastern mode of authoritarian state control may one day soon be capable of challenging Western capitalism in efficient resource allocation and economic (read: human quality of life) outcomes. If an authoritarian regime is able to grow rich enough, they can certainly challenge the freedom of the West (though that freedom is and has been diminishing and attacked for hundreds of years). Beyond that critical point, we should expect international economic selection to favor authoritarianism. Perhaps, actually, that has been the force driving the assault on civil liberties and personal and economic freedoms in the West ever since their introduction but that is far beyond our current scope of investigation.

The crux

The key issue of our time derives from the theory of the firm and the economics of governance. We can apply the lens of Coase, Williamson, and others to diagnose our present situation and guide our path forward. The central question is, if the market and free exchange with floating prices are the optimal means of governing the interactions between firms, why are firms internally governed hierarchically, at what point does hierarchy surpass market in efficacy, and upon which determinants does the location of that critical point most depend. In his Economics of Governance, Williamson notes of our present goal, “the object is to work out the efficiency logic for managing transactions by alternative modes of governance—principally spot markets, various long-term contracts (hybrids), and hierarchies.”

The two drivers of the choice of governance mode are uncertainty and transaction costs, of which the chief candidate for our present concern and the aspect that Williamson credits with the bulk of the explanatory power of the theory of the economics of governance is asset specificity. Asset specificity refers to the degree to which assets invested in by either party to a contract would require costly refactoring in order to be made productive in a similar arrangement with another firm. Uncertainty about the future requires investment in additional monitoring, verification, and commitment devices due to the impossible task of foreseeing and accounting for every eventuality in the initial contract.

We see the problem of large firm dominance, evincing the greater suitability of hierarchical instead of market governance, primarily in technology companies. As technology permeates the economy more broadly and all industries become increasingly technology-driven, the worry is that the entire economy will drift toward hierarchy over markets.

Let’s look at asset specificity and uncertainty in technology.

I have spent more time than I care to recount as a technologist and a leader of technologists railing against the specificity of our solutions and challenging my team and other leaders to build generic solutions. Unfortunately, our industry suffers untold delusions, chief among them the terrible triumvirate: 1. We need engineers, not computer scientists because we want technologists to “execute,” not invent; 2. Abstraction is the optional act of simplifying an implementation, not the act of inventing a new conceptual basis such that a change of basis from the current problem domain into this new conceptual framework results in a simpler or tractable problem; 3. Costs scale with the level of generality of a solution and are paid back only over time while every feature is business critical to have delivered yesterday.

I have always derided these three misconceptions as harmful to individual businesses. Now, I see them as the potential harbingers of the downfall of free and decentralized societies.

Because of these failures, nearly every solution built by corporate technology companies is specially-tailored to the task at hand. In fact, many “software engineers” pride themselves on never prematurely genericizing their solutions, giving in to the naïve empiricism that holds that we cannot know or predict anything about the future so we can only imagine a generic solution by looking at the union of the contours of existing specific solutions. Having built numerous point solutions, any integration with a new party or carving out of some existing set of functionality is a Herculean effort. In short, every software asset is highly specific and, therefore, incurs very high inter-firm transaction costs.

One particular asset, however, bears special mention. More and more, a company’s data is its most valuable asset as that data can inform predictive models of future behavior and predicting the future is always and everywhere valuable. After decades of formal research and fantastic generic solutions like RDF, data remains a mess of hacks and specificity. Integrating two datasets may well be impossible, depending on the entities represented and care taken in choosing identifiers. As datasets proliferate, just understanding the semantics of the data and cataloging the data available becomes a charter for a large team. Lacking standards for interoperability that are actually used in the real world, joining data across firms and building models on it is immensely more difficult than building models off of a single firm’s data. There are some Schelling points to be found in data structure but essentially none in data semantics. Transaction costs in data are enormous and growing with the increased uncertainty of the regulatory landscape concerning data privacy and the viability of cross-firm data sharing agreements, even with user consent and appropriate encryption, authorization, and anonymity.

Beyond asset specificity, the future of technology is as uncertain as other industries but the monitoring costs of verifying bona fide contractual performance are significantly higher. Software projects, undertaken by competent engineers, fail spectacularly at high rates, those that do succeed are often behind schedule or differ from their original specification, functionality is notoriously hard to specify, complexity and hence risk accrues exponentially with the addition of features and feature addition is absolutely required, small changes in function may require enormous changes in input (code), and small errors in input (code) can yield enormously large errors in function. These factors combine to often tilt the scales in favor of building (or vertically integrating) over buying except in exceptional circumstances where scope is narrow, specific and well known.

I believe that there are solutions to these problems and I believe that, if we don’t broadly adopt them, we are endangering the future freedom of humankind. Lacking a means of significantly reducing transaction costs for inter-firm software deals and a path towards credible contracting thereof, the gains to hierarchical organization writ large (i.e. authoritarianism) will continue to advance towards and eventually surpass those of decentralized, market-based interactions among free people.

I began this essay prior to the devolution of the United States into a civil war of a tyrannical central government and its vassal states violently suppressing the free speech and assembly of its supposedly free citizens. Reducing transaction and monitoring costs of software deals is a necessary but insufficient condition to restoring the primacy of the West. The re-ascendancy of the United States in a world of restored dominance of decentralized capitalism would require an incentive-compatible mechanism for restoring personal liberty and rolling back the creeping authoritarianism that now pervades life. While that is unlikely, we can take hope in two possibilities: perhaps this will be enough to tilt the balance in favor of freedom enough to stave off complete dominance by authoritarianism or perhaps some brave few will establish a properly free society, capable of profiting from capitalistic exchange among free people.

I will discuss solutions in many more articles and add links here.