Information theoretic measures for emergence and causality

The relationship between emergence and causation is contentious, with a long history. Most discussions are qualitative. Presented with a new system, how does one identify the microscopic and macroscopic scales that may be most useful for understanding and describing the system? Can Judea Pearl’s seminal ideas about causality be implemented practically for understanding emergence?

Broadly speaking, a weakness of discussions of emergence and causality is that it is hard to define these concepts in a rigorous and quantitative manner that makes them amenable to empirical testing, with respect to theoretical models and to experimental data. 

Fortunately, in the past decade, there have been some specific proposals to address this issue, mostly using information theory. A helpful recent review is by Yuan et al. 

“Two primary challenges take precedence in understanding emergence from a causal perspective. The first is establishing a quantitative definition of emergence, whereas the second involves identifying emergent behaviors or phenomena through data analysis.

To address the first challenge, two prominent quantitative theories of emergence have emerged in the past decade. The first is Erik Hoel et al.’s theory of causal emergence [19] whereas the second is Fernando E. Rosas et al.’s theory of emergence based on partial information decomposition [24].

Hoel et al.’s theory of causal emergence specifically addresses complex systems that are modeled using Markov chains. It employs the concept of effective information (EI) to quantify the extent of causal influence within Markov chains and enables comparisons of EI values across different scales [19,25]. Causal emergence is defined by the difference in the EI values between the macro-level and micro-level."

One perspective on causal emergence is that it occurs when the dynamics of a system at the macro-level is described more efficiently by macro-variables than by the dynamics of variables from the micro-level.

Klein et al. used Hoel’s information-theoretic measures of causal emergence to analyse protein interaction networks (interactomes) in over 1800 species, containing more than eight million protein–protein interactions, across different scales. They showed the emergence of ‘macroscales’ that are associated with lower noise and uncertainty. The nodes in the macroscale description of the network are more resilient than those in less coarse-grained descriptions. Greater causal emergence (i.e., a stronger macroscale description) was generally seen in multicellular organisms compared to single-cell organisms. The authors quantified causal emergence in terms of mutual information (between large and small scales) and effective information (a measure of the certainty in the connectivity of a network). Philip Ball (2023) (pages 218-220) gives an account of this work in terms of the emergence of multicellularity in biological evolution. He introduced the term causal spreading (pages 225-7), arguing that over the history of evolution the locus of causation has changed.

Yuan et al. continue

"However, in Hoel’s theory of causal emergence, it is essential to establish a coarse-graining strategy beforehand. Alternatively, the strategy can be derived by maximizing the effective information (EI) [19]. However, this task becomes challenging for large-scale systems due to the computational complexity involved. To address these problems, Rosas et al. introduced a new quantitative definition of causal emergence [24] that does not depend on coarse-graining methods, drawing from partial information decomposition (PID)-related theory. PID is an approach developed by Williams et al., which seeks to decompose the mutual information between a target and source variables into non-overlapping information atoms: unique, redundant, and synergistic information [29]…"

The Figure below is taken from Rosas et al. Xt^j (j=1,…,n) are microscopic variables that define a Markov chain. Vt is a macroscopic variable that is completely determined by the microscopic variables.

“Diagram of causally emergent relationships. Causally emergent features have predictive power beyond individual components. Downward causation takes place when that predictive power refers to individual elements; causal decoupling when it refers to itself or other high-order features.”

Rosas et al. applied the method to specific systems, including Conway’s Game of Life, Reynolds’ flocking model, and neural activity as measured by electrocorticography. More recently, it was used to describe emergence in computer science, including the identification of modular structures. Calculations were performed for specific examples, including Ehrenfest’s urn model for diffusion, the Ising model with Glauber dynamics, a Hopfield neural network model for associative memory.

Yuan et al. also state the following:

"The second challenge pertains to the identification of emergence from data. In an effort to address this issue, Rosas et al. derived a numerical method [24]. However, it is important to acknowledge that this method offers only a sufficient condition for emergence and is an approximate approach. Another limitation is that a coarse-grained macro-state variable should be given beforehand to apply this method."

Sas et al. recently stated

“Empirical applications of this framework to study emergence … including the study of gene regulatory networks [22], the dynamics of the human brain [23], the internal dynamics of reservoir computing [24], and the formation of useful internal representations in machine learning [25].”

Yuan et al. also discuss two significant connections between causal emergence and machine learning. First, machine learning can be used to improve calculations of causal emergence. Second, causal emergence measures can be used to better understand how machine learning works and improve it.

The work described above built on earlier work by Crutchfield, who claimed that the identification of emergence and hierarchies could be made operational, stating that “different scales are delineated by a succession of divergences in statistical complexity at lower levels.” More recently, Rupe and Crutchfield have reported progress towards identifying emergent self-organisation in a system.

Although this work on quantitative measures of emergence based on information theory represents significant progress, there are many open problems. Examples include the extension to non-Markovian systems and the development of computationally feasible methods for large systems. The latter is particularly important in physical systems where spontaneous symmetry breaking occurs, as this only happens in the thermodynamic limit of an infinite system.

There is an unrecognised similarity between the work described above and techniques recently developed to characterise phase transitions in statistical mechanics models such as the Ising model and classical dimer models. Coarse-graining (CG) is optimised by maximising the Real-Space Mutual Information (RSMI) between a spatial block and its distant environment. 

In general, maximising mutual information is notoriously hard but can be done using state-of-the-art machine learning algorithms. Gokmen et al. have developed an algorithm that they claim “can, unsupervised, construct order parameters, locate phase transitions, and identify spatial correlations and symmetries for complex and large-dimensional real-space data.” Furthermore, the optimal CG explicitly identifies the scaling operators associated with the critical point. 

The classical dimer model provides a stringent test as “the relevant low-energy degrees of freedom are profoundly different from the microscopic building blocks of the theory and change qualitatively throughout the phase diagram.” In other words, the emergent entities (quasiparticles such as vortices associated with the height field, which is described by a sine-Gordon field theory) are different from the dimers.

It is encouraging to see that two different scientific communities have developed similar ideas to address this challenging problem of making discussions about emergence and causality more concrete and quantitative.

Biology is about emergence in subtle ways

Biology is a field that is all about emergence. It exhibits a hierarchy of structures from DNA to proteins to cells to organs to organisms. Phenotypes emerge from genotypes. At each level of the hierarchy (stratum) there are unique entities, phenomena, principles, methods, theories, and sub-fields. But there is more to the story. 

Philip Ball is probably my favourite science writer. Earlier this year, he gave a beautiful lecture at The Royal Institution, What is Life and How does it Work?

The lecture presents the main ideas in his recent book,  How Life Works: A User's Guide to the New Biology

Here are a few things that stood out for me from the lecture.

1. The question, "What is life?" has been and continues to be notoriously difficult to answer.

2. The Central Dogma of Molecular Biology is wrong.

It was originally stated by Francis Crick, and some commonly assumed corollaries of it are wrong. In simple terms, the Dogma states that DNA makes RNA and RNA makes proteins. This is a unique and unidirectional process. For example, a specific code (string of the letters A,G,T, and C) will produce a specific protein (sequence of amino acids) which will naturally fold into a unique structure with a specific biochemical function. 

The central dogma has undergirded the notion that genes determine everything in biology. Everything is bottom-up.

However, Ball gives several counterexamples.

A large fraction of our DNA does not code for proteins.

Many proteins are disordered, i.e., they do not have a unique folded structure.

Aside: An earlier failure of (some versions of) the central dogma was the discovery of reverse transcriptase by the obscure virus club, essential for the development of HIV drugs and covid-19 vaccines.

3. The role of emergence can be quantified in terms of information theory, helping to understand the notion of causal emergence: the cause of large-scale behaviour is not just a sum of micro-causes, i.e., the properties of and interactions between the constituents at smaller scales. Entities at the level of the phenomena are just as important as what occurs at lower levels.

(page 214 in the book). Causal emergence is concerned with fitting the scale of the causes to the scale of the effects.

The figure above is taken from this paper from 2021.

Evolution and emergence: higher order information structure in protein interactomes across the tree of life 

Brennan Klein, Erik Hoel, Anshuman Swain, Ross Griebenow, Michael Levin

The authors quantify casual emergence in protein networks in terms of mutual information (between large and small scales) and effective information (a measure of the certainty in the connectivity of a network).

Aside: These quantitative notions of emergence have been developed more in recent work by Fernando Rosas and collaborators and discussed in a Quanta article by Philip Ball.

4. Context matters.  A particular amino acid sequence does not define a unique protein structure and function. They may depend on the specific cell in which the protein is contained.

5. Causal spreading.  Causality happens at different levels. It does not always happen at the bottom (genetic level). Sometimes it happens at higher levels. And, it can flow up or down.

6. Levels of description matter. This is well illustrated by morphology and the reasons that we have five fingers. This is not determined by genes.

7. Relevance to medicine. There has been a focus on the genetic origin of diseases. However, many diseases, such as cancer, do not predominantly happen at the genetic level. There has been a prejudice to focus on the genetic level, partly because that is where most tools are available. For cancer, focussing on other levels, such as the immune system, may be more fruitful.

8. Metaphors matter. Biology has been dominated by  metaphors such as living things are "machines made from genes" and "computers running a code". However, metaphors are metaphors. They have limitations, particularly as we learn more. All models are wrong, but some are useful. Ball proposes that metaphors from life, including the notion of agency, may be more fruitful.

9. The wisdom of Michael Berry. Ball ends with Berry's saying that the biggest unsolved problem in physics is not about dark matter (or some similar problem), but rather, "If all matter can be described by quantum theory, where does the aliveness of living things come from?" In other words, "Why is living matter so different from other matter?"

There is also an interesting episode of the How To Academy podcast, where Ball is interviewed about the book.

Clarity about the relationship of emergence, complexity, predictability, and universality

Emergence means different things to different people. Except, that practically everyone likes it! Or at least, likes using the word. Terms associated with emergence include novelty, unpredictability, universality, stratification, and self-organisation. We need to be clearer about what we mean by each of these terms and how they are related or unrelated. Significant progress is reported in a recent preprint.

Software in the natural world: A computational approach to hierarchical emergence

Fernando E. Rosas, Bernhard C. Geiger, Andrea I Luppi, Anil K. Seth, Daniel Polani, Michael Gastpar, Pedro A.M. Mediano

This preprint is the subject of a nice article in Quanta Magazine.

The New Math of How Large-Scale Order Emerges by Philip Ball

Ball defines emergence in terms of unpredictability. He states: 

"Loosely, the behavior of a complex system might be considered emergent if it can’t be predicted from the properties of the parts alone."

He describes the work of Rosas et al. as follows, 

"A complex system exhibits emergence, according to the new framework, by organizing itself into a hierarchy of levels that each operate independently of the details of the lower levels."

This is defining emergence in terms of universality. Rosas et al. use an analogy with software, which runs independently of the details of the hardware of the computer and does not depend on microscopic details such as electron dynamics.

There are three types of closure associated with emergence: informational, causal, and computational.

Informational closure means that to predict the dynamics of the system at the macroscale one does not need any additional  information from the microscale.

Equilibrium thermodynamics is a nice example. 

Causal closure means that the system can be controlled at the macroscale without any knowledge of lower-level information.

"Interventions we make at the macro level, such as changing the software code by typing on the keyboard, are not made more reliable by trying to alter individual electron trajectories."

"...we can use macroscopic variables like pressure and viscosity to talk about (and control) fluid flow, and knowing the positions and trajectories of individual molecules doesn’t add useful information for those purposes. And we can describe the market economy by considering companies as single entities, ignoring any details about the individuals that constitute them."

Computational closure is a more technical concept. 

"a conceptual device called the ε-(epsilon) machine. This device can exist in some finite set of states and can predict its own future state on the basis of its current one. It’s a bit like an elevator, said Rosas; an input to the machine, like pressing a button, will cause the machine to transition to a different state (floor) in a deterministic way that depends on its past history — namely, its current floor, whether it’s going up or down and which other buttons were pressed already. Of course an elevator has myriad component parts, but you don’t need to think about them. Likewise, an ε-machine is an optimal way to represent how unspecified interactions between component parts “compute” — or, one might say, cause — the machine’s future state."

Aside: epsilon-machines featured significantly in my previous post about What is a complex system? 

"Computational mechanics allows the web of interactions between a complex system’s components to be reduced to the simplest description, called its causal state."

"...for an emergent system that is computationally closed, the machines at each level can be constructed by coarse-graining the components on just the level below: They are, in the researchers’ terminology, “strongly lumpable.”"

In some sense, this may be related to the notion of quasiparticles and effective interactions in many-body physics. 

Aside: In 1962, Herbert Simon identified hierarchies as an essential feature of complex systems, both natural and artificial. A key property of a level in the hierarchy is that it is nearly decomposable into smaller units, i.e., it can be viewed as a collection of weakly interacting units. The time required for the evolution of the whole system is significantly decreased due to the hierarchical character. The construction of an artificial complex system, such as a clock, is faster and more reliable if different units are first assembled separately and then the units are brought together into the whole. Simon argues that the reduction in time scales due to modularity is why biological evolution can occur on realistic time scales.  The 1962 article is reprinted in The Sciences of the Artificial.

The paper by Rosas et al. is one of the most important ones I have encountered in the past few years. I am slowly digesting it.

The beauty of the paper that it is mathematically rigorous. All the concepts are precisely defined and the central results are actually theorems. This replaces the vagueness of most discussions of emergence, including by myself.

The paper has helpful figures and considers concrete examples including Ehrenfest's Urn, an Ising model with Glauber dynamics, and a Hopfield neural network model.

I thank Gerard Milburn for bringing the Quanta article to my attention.

The challenge of writing books on water

Biman Bagchi has just published a new book,
Water in Biological and Chemical Processes: From Structure and Dynamics to Function 

Cambridge University Press sent me a complimentary copy to review. I am slowly working through it and will write a detailed review when I am done.

I think this is a very challenging subject to write a book on for at least three reasons. First, the scope of the topic is immense. Furthermore, it is multi-disciplinary spanning physics, chemistry, and biology, with a strong interaction between experiment, theory, and simulation. Second, although there have been some significant advances in the last few decades there is real state of flux, with a fair share of controversies, advances, and fashions. Finally, which audience do you write for? Experimental biochemists or theoretical physicists or somewhere in between.

Although this is an incredibly important and challenging topic few authors have taken up the challenge. One who has is Arieh Ben-Naim

Molecular Theory of Water and Aqueous Solutions, Part I: Understanding Water (2009)

Molecular Theory of Water and Aqueous Solutions Part II: The Role of Water in Protein Folding, Self-Assembly and Molecular Recognition (2011)

This was a topic of great interest to my late father. He wrote two comprehensive reviews with John Edsall, published in Advances in Biophysics

Water and proteins. I. The significance and structure of water; its interaction with electrolytes and non-electrolytes (1977) [does not seem to be available online]

Water and proteins. II. The location and dynamics of water in protein systems and its relation to their stability and properties (1983)

Classic earlier books include:

The Structure and Properties of Water
 by David Eisenberg and Walter Kauzmann
(1969, reissued in 2002 by Oxford UP in their Classic Texts in the Physical Sciences)

A seven volume series, Water: A comprehensive treatise, edited by Felix Franks

At the popular level there is
Life's Matrix: A Biography of Water 
(2001) by Philip Ball

Why does liquid water have a relatively high boiling point?

The graph below shows the boiling points of different molecular liquids of the chemical composition HnX. Note that at room temperature all of them, except H2O would be vapour.

What is so different about water?
It can form much stronger hydrogen bonds than the heavier ones, because X=O is the most electronegative.
H2O has two proton donors and two lone pair orbitals which act as acceptors for hydrogen bonds. Hence, each molecule can form four hydrogen bonds in a tetrahedrally co-ordinated liquid.
In contrast NH3 has three donors and one acceptor. Hence, it can only form two hydrogen bonds with neighbours.

This can all be illustrated with some very cute pictures of little men grabbing hands and legs in Philip Ball's book H2O: a biography of water.

I first saw all this nicely explained by Jim Skinner a few months ago in a Town Talk at the Telluride Science Research Center in Colorado.

Quantum biology? Tunneling in enzymes

Over the past two decades the possibility of quantum tunneling of protons in enzymes
has attracted considerable attention. (See for example a piece in Nature by Philip Ball (my favourite science writer) or the proceedings of a meeting at the Royal Society

The observed large kinetic isotope effects and their temperature dependence are inconsistent with semi-classical transition state theory, whereby the chemical reaction occurs via thermal activation over an energy barrier. These discrepancies have been interpreted as evidence for the presence of tunneling. However, it should be stressed that this evidence is indirect, being based on the values of fitting parameters for Arrhenius plots where the absolute temperature only varies by about ten per cent.
In contrast, for chemical reactions involving simpler organic molecules, such as benzoic acid much more definitive signatures of proton tunneling have been observed. These include a temperature independent rate at low temperatures and tunnel splitting of the ground
state energy.
Key questions that need to be answered include:

• Can all the known experimental results be explained without tunneling?
• To what extent is it necessary to go beyond the traditional semi-classical transition state theory to explain the observed kinetic isotope effects of enzymes?
• If tunneling does occur, is it actually important for the function of the enzyme?
• Have enzymes evolved in a manner that enhances the contribution of tunneling?

There are currently a wide range of views on the answers to these questions.
For example, a review in Science, How Enzymes Work: Analysis by Modern Rate Theory and Computer Simulations by Mireia Garcia-Viloca, Jiali Gao, Martin Karplus, and Donald G. Truhlar states that,

``the entire and sole source of the catalytic power of enzymes is due to the lowering of the free energy of activation and any increase in the generalized transmission co-efficient, as compared to that of the uncatalyzed reaction."

Villa and Warshel state that,

``the most important contribution to catalysis comes from the reduction of the activation free energy by electrostatic effects ... the popular proposal that enzymes catalyze reactions by special dynamical effects is not supported by a consistent simulation study ... the interpretation of recent experiments as evidence for dynamical contributions to catalysis is unjustified.''

In contrast, Klinman and collaborators stated in a 1999 Nature paper that,

``Our present findings on hydrogen transfer under physiological conditions cannot be explained without invoking both quantum mechanics and enzyme dynamics.''

In a paper that focused on simulations Schwartz and collaborators express a similar view,

``The action of the enzyme in speeding the chemical reaction, however, is postulated to be intimately connected to the directed vibrational motion identified in this paper. Thus, it appears that evolution has designed the protein matrix of an enzyme not just to hold substrates or stabilize transition state formation, but rather to channel energy in a specific chemically relevant direction.''

I align myself with the skeptics in a paper I am working on..... more to come..