Computer scientists and engineers
A part of the solution to The Greatest Challenge of our Time.
I remember, not so long ago, when we called what we did “computer science” and our department was named “research and development.” Granted, this was at a company that, while one of the most successful software companies in history, isn’t traditionally considered in the “big tech” club and they’ve since changed their department name to “engineering” to match the behemoths with whom they compete for “talent.” It was a sad day that their hand was so forced. Words matter and our industry suffers from a lack of computer scientists and a surfeit of so-called engineers.
The work of software is special. The real world imposes constraints that localize effects. Complexity surely exists but inverse-square laws ensure that, though chains of causality may be long, circuitous, and magnifying, we typically understand the area of influence if not the immediate effects of some action. On top of that, we have had a few thousand years to begin to amass a cataloging and, more recently, an understanding of the various mechanisms by which some parts in the real world can effect other parts and how those parts tend to interact to form wholes.
In software, we begin tabula rasa, unconstrained by anything but our own ingenuity and desire. And from this blank canvas, driven by some desired outcome, we begin to construct towering complexity unmatched in the real world. In the real world of atoms, with the constraints of inverse-square laws, the typical scales at which we operate, and restricted dimensionality, there is a real limit on the degree of interactions and, therefore, unintended and unforeseen consequences of a change to a part on the whole. Not so with software. We have no limit of dimensionality and no built-in inverse-square analogs constraining the mess we can make. In its “natural state,” software effects are global, not localized.
Turning from implementation to desired outcome, we are again unconstrained: we are capable of building anything we can dream up in the computable universe. And so, we imagine up untold complexities, handling untold edge cases, satisfying as many disconnected “user stories” as possible. And then we try to build the thing, agilely, feature by vertically-sliced feature. And then we wonder why it’s hard to add that next feature or why project 3,783 broke that thing we delivered in project 2,964.
The core problem, in product as well as software building (and multiplied when commingled), is that you can’t build up from nothing and expect to end up with a coherent whole.
Whether in designing software products or building software systems, we first have to build (or discover) down and only then can consider building up.
In the real world, the job of science is to posit and test the underlying structure that generates observed phenomena (wholes). That is, the job of the scientist is to build or discover downward from phenomena to structure, from wholes to parts. The complex systems community will argue that this is reductionist and so it is but reductionism has been wildly successful in an incredibly diverse set of domains and it still represents the first step towards understanding complex domains because, without an understanding of the components of your system and their relations, there is no hope of analyzing the system as a whole.
The job of the engineer then becomes assembling the scientist-discovered components (conceptual or physical) into wholes intended to achieve some function. Namely, building ”up” from components to wholes, structure to phenomena. Engineers have certainly run ahead of scientists at times and we still have items that engineers have built that scientists can’t explain yet these are primarily instances of emergent effects serendipitously stumbled upon through trial and error of various configurations of parts into wholes. The assembling entails enormous complexity in many cases as the relation between parts can create unexpected effects on the whole. The matter remains that, absent an understanding of the parts and their relations, we are rudderless in pursuit of some desired wholes, though we may occasionally drift upon a propitious shore.
Now, we return to software.
In software—at least as practiced at “big tech” today—we suffer a dearth of scientists. Not only do we lack people interested in “building down” but we actively disincentivize the activity, contributing to a culture that scoffs at anything to which it can pin the monicker of “big design up front.” We create misguided “rules of 3,” intended to be used as a cudgel to stop any engineer from “prematurely” abstracting their work. We throw around absurd acronyms like YAGNI (you ain’t gonna need it) to justify labeling generalization and anticipating future system evolution as evils instead of the obvious goods they are. All of these are symptoms of a reaction against the core misunderstanding of abstraction in software: abstraction isn’t primarily an implementation concern, it’s a design concern. That is, the “rule of 3” assumes that abstraction is meant to avoid repetition in code, that it is a compression of the output of the engineer. Rather, abstraction is a compression of and set of constraints imposed upon the input of engineers, namely, the conceptual framework that they manipulate and implement; it just happens to typically come with the indirect benefit of compressing engineer output. When applied to outputs however, as is implied by the “rule of 3” and YAGNI, “abstraction” is accidental and represents only the joining together of code that looks similar rather than the creation of a new concept that drives product thinking, is often deserving of definition to users, and provides structure and constraints for further implementation. Without any scientists aiming at uncovering these true conceptual abstractions, which we can call essential abstractions, any “abstractions” arising from a desire to compress lines of code are purely accidental abstractions and may actually combine concepts that have no underlying connection. Hence the general aversion to accidental abstraction, though never previously distinguished from essential abstraction to my knowledge. None of the typical objections to accidental abstraction apply at all to essential abstraction.
The process of product and system design must begin with decomposition of the problem space into a vocabulary of abstractions and their interrelations, a conceptual framework that will provide the scaffolding upon which the product and the system that embodies it must be built. This is the job of a scientist, not a engineer.
The best engineers I have encountered have been considered great because they actually fill the role of scientist instead of engineer. They provide the raw material that all of the other engineers recompose into the features they’re building but, equally importantly, their insight into the true nature of the problem space exerts strong influence back onto the definition of the product itself, reframing the manner in which business, product, and design teams think about their customers and product. That is, with the scientist role filled, not only does desired phenomena drive implementation but the process of implementation drives further desired phenomena.
We must start building software by defining the phenomena we wish to generate: the set of functionality we wish to provide our users. Instead of then diving directly into an effort of blindly building the functionality we defined, we should recognize that working backwards from phenomena is, very fundamentally, the job of scientists. We then explore our phenomena, break it into conceptual components, recompose them experimentally, getting a feel for their relations and whether the generalized set of compositions is closed under the criteria that they should produce acceptable phenomena. Once we have this core set of user-visible abstractions, we can then begin to implement each and the allowed relations between them in a real system.
This process doesn’t remove all complexities of implementation but it does have many benefits.
- Everyone from users to product managers to designers to builders speak the same language and manipulate the same concepts. This drives cohesion and clarity of mission, increases the likelihood of actually stumbling upon good ideas, and, most importantly, yields software users can understand.
- The embodiment of the product in code more closely aligns to user-relevant concepts, which allows for a much easier mapping of product changes onto the software system. Whenever possible, reify. By first building down, we enable reification by having previously identified and distinguished.
- Built on a scaffold of properly thought-out and defined concepts with explicit relations, product features are uncovered and evolved rather than shoehorned and features have more natural implementations.
In the standard model of big tech projects, with thousands of point features to add in a nearly arbitrary order, of course you need thousands of engineers carefully glomming changes together throughout a mess of a system to approximate the desired phenomena. Without a coherent conceptual framework driving the product, of course the software system that embodies it faces a significant mismatch with its user-visible features, which drives additional accidental complexity in the implemented system and requires concomitant hoards of engineers to manage it. Most importantly, because of the inevitable entanglement of features and misalignment between the product and the software system, an enormous amount of effort needs to be expended to separate out any aspect. Incessant re-orgs and constant renegotiation of mission and system ownership are the immediate symptoms of a product without conceptual scaffolding. In such an environment, where a single feature maps to slices of dozens of system components, mission boundaries are transitory because no obvious grouping of product initiatives exists, every feature investment is highly asset-specific and intra-company management is the only reasonable governance.
Alternatively, a product developed from a coherent conceptual framework has clear component boundaries, immediately admitting to development by separate entities with other entities potentially managing their composition. If organized internally, teams are long-lived and fully mission-oriented, re-orgs scarce, and, by nature, intra-team work dominates inter-team communication overhead.
Investing in science, in building and discovering down before asking engineers to build up, yields huge advantages in usability of the product, adaptability and ease of implementation, scope and mission of teams, and the degree of complexity attainable for any given level of engineering investment. It just happens to have the added kicker of enabling smaller companies to outcompete behemoths.
Though stated a bit differently, the idea of building down to a set of concepts before building up is a clear homage to Rich Hickey’s hammock time spent decomplecting the aspects of a problem. His talks are the definitive reference.
Ryan Singer recently gave a fantastic primer on Christopher Alexander’s ideas about form and function and it struck me as incredibly relevant to our present discussion (I ordered the books and look forward to reading more). The conceptual components I discuss seem akin to Alexander’s centers and the idea of outlining the centers and their relations first, then making finer-grained plans when situated seems to subsume much of this thinking.