What is a complex system?

What do we mean when we say a particular system is "complex"? We main have some intuition that it means there are many degrees of freedom and/or that it is hard to understand. "Complexity" is sometimes used as a buzzword, just like "emergence." There are many research institutes that claim to be studying "complex systems" and there is something called "complexity theory". Complexity seems to mean different things to different people.

I am particularly interested in understanding the relationship between emergence and complexity. To do this we first need to be more precise about what we mean by both terms. A concrete question is the following. Consider a system that exhibits emergent properties. Often that will be associated with a hierarchy of scales. For example: atoms, molecules, proteins, DNA, genes, cells, organs, people. The corresponding hierarchy of research fields is physics, chemistry, biochemistry, genetics, cell biology, physiology, psychology. Within physics a hierarchy is quarks and leptons, nuclei and electrons, atoms, molecules, liquid, and fluid. 

In More is Different, Anderson states that as one goes up the hierarchy the system scale and complexity increases. This makes sense when complexity is defined in terms of the number of degrees of freedom in the system (e.g., the size of the Hilbert space needed to describe the complete state of the system). On the other hand, the system state and its dynamics become simpler as one goes up the hierarchy.  The state of the liquid can be described completely in terms of the density, temperature, and the equation of state. The dynamics of the fluid can be described by the Navier-Stokes equation. Although that is hard to solve in the regime of turbulence, the system is still arguably a lot simpler than quantum chromodynamics (QCD)! Thus, we need to be clearer about what we mean by complexity.

To address these issues I found the following article very helpful and stimulating.

What is a complex system? by James Ladyman, James Lambert, and Karoline Wiesner 

It was published in 2013 in a philosophy journal, has been cited more than 800 times, and is co-authored by two philosophers of science and a physicist.

[I just discovered that Ladyman and Wiesner published a book with the same title in 2020. It is an expansion of the 2013 article.].

In 1999 the journal Science had a special issue that focussed on complex systems, with an Introduction entitled, Beyond Reductionism. Eight survey articles covered complexity in physics, chemistry, biology, earth science, and economics.

Ladyman et al., begin by pointing out how each of the authors of these articles chooses different properties to define what complexity is associated. These characteristics include non-linearity, feedback, spontaneous order, robustness and lack of central control, emergence, hierarchical organisation, and numerosity.

The problem is that these characteristics are not equivalent. If we do choose a specific definition for a complex system, the difficult problem then remains of determining whether each of the characteristics above is necessary, sufficient, both, or neither for the system to be complex (as defined). This is similar to what happens with attempts to define emergence.

Information content is sometimes used to quantify complexity. Shannon entropy and Kolmogorov complexity (Sections 3.1, 3.2) are discussed. The latter is also known as algorithmic complexity. This is the length of the shortest computer program (algorithm) that can be written to produce the entity as output. A problem with both these measures are they are non-computable.

Deterministic complexity is different from statistical complexity (Section 3.3). A deterministic measure treats a completely random sequence of 0s and 1s as having maximal complexity. A statistical measure treats a completely random sequence as having minimal complexity. Both Shannon and algorithmic complexity are deterministic.

Section 4 makes some important and helpful distinctions about different measures of complexity.

3 targets of measures: methods used, data obtained, system itself

3 types of measures: difficulty of description, difficulty of creation, or degree of organisation

They then review three distinct measures that have been proposed logical depth (Charles Bennett), thermodynamic depth (Seth Lloyd and Heinz Pagels), and effective complexity (Murray Gell-Mann).

Logical depth and effective complexity are complementary quantities. The Mandelbrot set is example of a system (set of data) that exhibits a complex structure that has a high information content. It is difficult to describe. It has a large logical depth.

Created by Wolfgang Beyer with the program Ultra Fractal 3. 

On the other hand, the effective complexity of the set is quite small since it can be generated using the simple equation

z_n+1 = c + z_n^2

c is a complex number and the Mandelbrot set is the values of c for which the iterative map is bounded.

Ladyman et al, prefer the definition of a complex system below, but do acknowledge its limitations.

(Physical account) A complex system is an ensemble of many elements which are interacting in a disordered way, resulting in robust organisation and memory. 

(Data-driven account) A system is complex if it can generate data series with high  statistical complexity. 

What is statistical complexity? It relates to degrees of pattern and some they refer to as causal state reconstruction. It is applied to data sets, not systems or methods. Central to their definition is the idea of the epsilon-machine, something introduced in a long and very mathematical article from 2001, Computational Mechanics: Pattern and Prediction, Structure and Simplicity, by Shalizi and Crutchfield.

The article concludes with a philosophical question. Do patterns really exist? This relates to debates about scientific realism versus instrumentalism. The authors advocate something known as "rainforest realism", that has been advanced by Daniel Dennett, Don Ross, and David Wallace. A pattern is real if one can construct an epsilon-machine that can simulate the phenomena and predict its behaviour.

I don't have a full appreciation or understanding of where the article ends up. Nevertheless, the journey there is helpful as it clarifies some of the subtleties and complexities (!) of trying to be more precise about what we mean by a "complex system".

A few things condensed matter physics has taught me about science (and life)

We all have a worldview, some way that we look at life and what we observe. There are certain assumptions we tend to operate from, often implicitly. Arguably, our worldview is shaped by our experiences: family, friendships, education, jobs, community organisations, and our cultural context (political, economic, and social).

A significant part of my life experience has been working in universities as a condensed matter physicist and being part of a broader scientific community. Writing a Condensed Matter Physics: A Very Short Introduction crystallised some of my thoughts about what CMP might mean in broader contexts. I am more aware of how my experience in CMP has had a significant influence on the way I view not just the scientific enterprise, but also broader philosophical and social issues. Here are a few concrete examples.

Complex systems. The objects studied in condensed matter physics have many interacting components (atoms). Further, there is an incredible diversity of systems (materials and phenomena) that are studied. Many different properties and parameters are needed to characterise a system and its possible states. There are many different ways of investigating each system. Similarly, almost everything else of interest in science and life is a complex system.

Emergence. This is central to CMP. The whole is greater than the sum of the parts. The whole is qualitatively different from the parts. Related features include robustness, universality, surprises, and the difficulty of making predictions. An emergent perspective can provide insights into other complex systems: from biology to psychology to politics.

Differentiation and integration. A key aspect of describing and understanding a complex system is conceptually breaking it into smaller parts (differentiation), determining how those parts interact with one another, and determining how those interacting parts combine to produce properties of the whole system (integration).

Diversity: The value of multiple perspectives and methods. Due to the complexity of condensed matter systems, multiple methods are needed to characterise their different properties. Due to emergence, there are various scales and hierarchies present. Investigating and describing the system at these different scales provides different perspectives on the system. What does the scientist do with all these different perspectives? Interpretation and synthesis are needed. That is not an easy or clearcut enterprise.

 Navigating the middle ground. The most interesting CMP occurs in an intermediate interaction regime that is challenging theoretically. Insight can be gained by considering two extremes that are more amenable to analysis: weak interaction and strong interaction. I had fun using conservative-liberal political tensions as a metaphor for divisions in the strongly correlated electron community.

The Art of Interpretation. Everything requires interpretation: a phone text message, a newspaper article, a novel, a political event, data from a science experiment, and any scientific theory. With interpretation, we assign meaning and significance to something. How we do this is complex and draws on our worldview, both explicitly and implicitly. Regardless of our best intentions, interpretation always has subjective elements.

Synthesis. Given the diversity of data, perspectives, and interpretation, it is a challenge to synthesise them into some coherent and meaningful whole. All the pieces are rarely consistent with one another. Some will be ignored, some discarded, some considered peripheral, and others central. This synthesis is also an act of interpretation.

All models are wrong but some are useful. One way to understand complex systems is in terms of "simple" models that aim to capture the essential features of certain phenomena. In CMP significant progress (and many Nobel Prizes) has resulted from the proposal and study of such models. There is a zoo of them. Many are named after their main inventor or proponent: Ising, Anderson, Hubbard, Heisenberg, Landau, BCS,... All theories in CMP are also models since they involve some level of approximation, at least in their implementation. These models are all wrong, in the sense that they fail to describe all features and phenomena of the system. But, the best models are useful. Their simplicity makes them amenable to understanding, mathematical analysis, or computer simulation. Furthermore, the models can give insight into the essential physics underlying phenomena, predict trends, or be used to analyse experimental data. 

The autonomy of academic disciplines. Reality is stratified. At each level of the hierarchy, one has unique phenomena, methods, concepts, and theories. Most of these are independent of the details of what happens at lower levels of the hierarchy. Given the richness at each level, I do not preference one discipline as more fundamental or important than the others.

Pragmatic limits to knowledge. We know so much.  We know so little. On the one hand, it is amazing to me how successful CMP has been. We have achieved an excellent understanding, at least qualitatively of many emergent phenomena in systems that are chemically and structurally complex (e.g., liquid crystals and superconductivity in crystals involving many chemical elements). On the other hand, there are systems such as glasses and cuprate superconductors that have been incredibly resistant to understanding. Good research is very hard, even for the brilliant. Gains are often incremental and small. This experience leads me to have sober expectations about what is possible, particularly as one moves from CMP to more complex systems such as human societies, national economies, and brains.

Science is a human endeavour. Humans can be clever, creative, insightful, rational, objective, cooperative, fiercely independent and capable of great things. The achievements of science are a great testimony to the human spirit. Humans can also be stubborn, egotistical, greedy, petty, irrational, ruthlessly competitive, and prone to fads, mistakes and social pressures. Science always happens in a context: social, political, cultural, and economic. Context does not determine scientific outcomes but due to human nature, it can corrupt how science is done.

The humanity of scientists leads to a lack of objectivity captured in Walter Kauzmann's maxim: people will tend to believe what they want to believe rather than what the evidence before them suggests that they should believe. My decades of experience working as a scientist leads me to scepticism about extravagant claims that some scientists make, particularly hype about the potential significance (scientific, technological, or philosophical) of their latest discovery or their field of research. Too often such claims do not stand the test of time.

Humility. This brings together practically everything above. The world is complex, people are complex, and human-world interactions are complex. It is easy to be wrong. We often have a pretty limited perspective of what is going on. 

Are gravity and spacetime really emergent in AdS-CFT?

There is an interesting Scientific American article by Adam Becker

What Is Spacetime Really Made Of?

Spacetime may emerge from a more fundamental reality. Figuring out how could unlock the most urgent goal in physics—a quantum theory of gravity

It considers two different approaches to quantum gravity (loop quantum gravity and AdS-CFT beloved by string theorists). Compared to some Scientific American articles it is moderately balanced and low on hype. The article has a nice engagement with some philosophers of physics. It is clear to me how loop quantum gravity has a natural interpretation that gravity and space-time are emergent. However, that is not clear for AdS-CFT.

 The following paragraph is pertinent.

But there are other ways to interpret the latest findings. The AdS/CFT correspondence is often seen as an example of how spacetime might emerge from a quantum system, but that might not actually be what it shows, according to Alyssa Ney, a philosopher of physics at the University of California, Davis. 

“AdS/CFT gives you this ability to provide a translation manual between facts about the spacetime and facts of the quantum theory,” Ney says. “That’s compatible with the claim that spacetime is emergent, and some quantum theory is fundamental.” 

But the reverse is also true, she says. The correspondence could mean that quantum theory is emergent and spacetime is fundamental—or that neither is fundamental and that there is some even deeper fundamental theory out there. Emergence is a strong claim to make, Ney says, and she is open to the possibility that it is true. “But at least just looking at AdS/CFT, I’m still not seeing a clear argument for emergence.”

What is really fundamental in science?

What do we mean when we say something in science is fundamental? When is an entity or a theory more fundamental or less fundamental than something else? For example, are quarks and leptons more fundamental than atoms? Is statistical mechanics more fundamental than thermodynamics? Is physics more fundamental than chemistry or biology? In a fractional quantum Hall state, are electrons or the fractionally charged quasiparticles more fundamental?

Answers depend on who you ask. Physicists such as Phil Anderson, Steven Weinberg, Bob Laughlin, Richard Feynman, Frank Wilczek, and Albert Einstein have different views.

In 2017-8, the Foundational Questions Institute (FQXi) held an essay contest to address the question, “What is Fundamental?” Of the 200 entries, 15 prize-winning essays have been published in a single volume. The editors give a nice overview in the Introduction.

This post is mostly about the essay, Fundamental? of the first prize winner, Emily Adlam, a philosopher of physics. She contrasts two provocative statements.

Fundamental means we have won. The job is done and we can all go home.

Fundamental means we have lost. Fundamental is an admission of defeat.

This raises the question of whether being fundamental is objective or subjective.

Examples are given from scientific history to argue that what is considered to be fundamental has changed with time. The reductionism has led to the drive to explain everything in terms of smaller and smaller entities, that are deemed 'more fundamental". But we find that smaller does not always mean simpler.

Perhaps we should ask what needs explaining and what constitutes a scientific explanation. For example, Adlam asks whether explaining the fact that the initial state of the universe had a low entropy [the "past hypothesis"] is really possible or should be an important goal.

She draws on the issue of the distinction between objective and subjective probabilities. Probabilities in statistical mechanics are subjective: they are a statement about our own ignorance about the details of the motion of individual atoms and not any underlying randomness in nature. In contrast, probabilities in quantum theory reflect objective chance.

as realists about science we must surely maintain that there is a need for science to explain the existence of the sorts of regularities that allow us to make reliable predictions... but there is no similarly pressing need to explain why these regularities take some particular form rather than another. Yet our paradigmatic mechanical explanations do not seem to be capable of explaining the regularity without also explaining the form, and so increasingly in modern physics we find ourselves unable to explain either. 

It is in this context that we naturally turn to objective chance. The claim that quantum particles just have some sort of fundamental inbuilt tendency to turn out to be spin up on some proportion of measurements and spin down on some proportion of measurements does indeed look like an attempt to explain a regularity (the fact that measurements on quantum particles exhibit predictable statistics) without explaining the specific form (the particular sequence of results obtained in any given set of experiments). But given the problematic status of objective chance, this sort of nonexplanation is not really much better than simply refraining from explanation at all. 

Why is it that objective chances seem to be the only thing we have in our arsenal when it comes to explaining regularities without explaining their specific form? It seems likely that part of the problem is the reductionism that still dominates the thinking of most of those who consider themselves realists about science

In summary, (according to the Editors) Adlam argues that "science should be able to explain the existence of the sorts of regularities that allow us to make reliable predictions. But this does not necessarily mean that it must also explain why these regularities take some particular form." 

we are in dire need of another paradigm shift. And this time, instead of simply changing our attitudes about what sorts of things require explanation, we may have to change our attitudes about what counts as an explanation in the first place. 

Here, she is arguing that what is fundamental is subjective, being a matter of values and taste.

In our standard scientific thinking the fundamental is elided with ultimate truth: getting to grips with the fundamental is the promised land, the endgame of science. 

She then raises questions about the vision and hopes of scientific reductionists. 

In this spirit, the original hope of the reductionists was that things would get simpler as we got further down, and eventually we would be left with an ontology so simple that it would seem reasonable to regard this ontology as truly fundamental and to demand no further explanation. 

But the reductionist vision seems increasingly to have failed. 

When we theorise beyond the standard model [BSM] we usually find it necessary to expand the ontology still more: witness the extra dimensions required to make string theory mathematically consistent.

It is not just strings. Peter Woit has emphasised how BSM theories, such as supersymmetry, introduce many more particles and parameters.

... the messiness deep down is a sign that the universe works not ‘bottom-up’ but rather ‘top-down,’ ... in many cases, things get simpler as we go further up.

Our best current theories are renormalisable, meaning that many different possible variants on the underlying microscopic physics all give rise to the same macroscopic physical theory, known as an infrared fixed point. This is usually glossed as providing an explanation of why it is that we can do sensible macroscopic physics even without having detailed knowledge of the underlying microscopic theories. 

For example, elasticity theory, thermodynamics and fluid dynamics all work without knowing anything about atoms, statistical mechanics, and quantum theory.

But one might argue that this is getting things the wrong way round: the laws of nature don’t start with little pieces and build the universe from the bottom up, rather they apply simple macroscopic constraints to the universe as a whole and work out what needs to happen on a more fine-grained level in order to satisfy these constraints.

This is rather reminiscent of Laughlin's views about what is fundamental.

Finally, I mention two other essays that I look forward to reading as I think they make particularly pertinent points.

Marc Séguin (Chap. 6) distinguishes "between epistemological fundamentality (the fundamentality of our scientific theories) and ontological fundamentality (the fundamentality of the world itself, irrespective of our description of it)."

"In Chap. 12, Gregory Derry argues that a fundamental explanatory structure should have four key attributes: irreducibility, generality, commensurability, and fertility."

[Quotes are from the Introduction by the Editors].

Some would argue that the Standard Model is fundamental, at least on some level. But it involves 19 parameters that have to be fixed from experiment. Related questions about the Fundamental Constants, have been explored in a 2007 paper by Frank Wilczek.

Again, I thank Peter Evans for bringing this volume to my attention.

Why do deep learning algorithms work so well?

I am interested in analogues between cognitive science and artificial intelligence. Emergent phenomena occur in both, there have been some fruitful cross-fertilisation of ideas, and the extent of the analogues is relevant to debates on fundamental questions concerning human consciousness.

Given my general ignorance and confusion on some of the basics of neural networks, AI, and deep learning, I am looking for useful and understandable resources.

Related questions are explored in a nice informative article from 2017 in Quanta magazine, New Theory Cracks Open the Black Box of Deep Learning by Natalie Wolchover.

Like a brain, a deep neural network has layers of neurons — artificial ones that are figments of computer memory. When a neuron fires, it sends signals to connected neurons in the layer above. During deep learning, connections in the network are strengthened or weakened as needed to make the system better at sending signals from input data — the pixels of a photo of a dog, for instance — up through the layers to neurons associated with the right high-level concepts, such as “dog.” 

After a deep neural network has “learned” from thousands of sample dog photos, it can identify dogs in new photos as accurately as people can. The magic leap from special cases to general concepts during learning gives deep neural networks their power, just as it underlies human reasoning, creativity and the other faculties collectively termed “intelligence.” 

Experts wonder what it is about deep learning that enables generalization — and to what extent brains apprehend reality in the same way.

The article describes work by Naftali Tishby and collaborators that provides some insight into why deep learning methods work so well. This was first described in purely theoretical terms in a 2000 preprint

The information bottleneck method, Naftali Tishby, Fernando C. Pereira, William Bialek 

The idea is that a network rids noisy input data of extraneous details as if by squeezing the information through a bottleneck, retaining only the features most relevant to general concepts.

Tishby was stimulated in new directions in

2014 after reading a surprising paper by the physicists David Schwab and Pankaj Mehta

 An exact mapping between the Variational Renormalization Group and Deep Learning 

[They] discovered that a deep-learning algorithm invented by Geoffrey Hinton called the “deep belief net” works, in a particular case, exactly like renormalization [group methods in statistical physics... When they]. applied the deep belief net to a model of a magnet at its “critical point,” where the system is fractal, or self-similar at every scale, they found that the network automatically used the renormalization-like procedure to discover the model’s state. 

Although this connection was a valuable new insight, the specific case of a scale-free system, is not relevant to many deep learning situations.

Tishby and Ravid Shwartz-Ziv discovered that 

Over the course of training, common patterns in the training data become reflected in the strengths of the connections, and the network becomes expert at correctly labeling the data, such as by recognizing a dog, a word, or a 1.

...layer by layer, the networks converged to the information bottleneck theoretical bound: a theoretical limit derived in Tishby, Pereira and Bialek’s original paper that represents the absolute best the system can do at extracting relevant information. At the bound, the network has compressed the input as much as possible without sacrificing the ability to accurately predict its label...

...deep learning proceeds in two phases: a short “fitting” phase, during which the network learns to label its training data, and a much longer “compression” phase, during which it becomes good at generalization, as measured by its performance at labeling new test data.

What these new discoveries teach us about the relationship between learning in humans and in machines is contentious and explored briefly in the article. Although neural nets were inspired by the structure of the human brain the connection with the neural nets used today is tenuous.

The mystery of how brains sift signals from our senses and elevate them to the level of our conscious awareness drove much of the early interest in deep neural networks among AI pioneers, who hoped to reverse-engineer the brain’s learning rules. AI practitioners have since largely abandoned that path in the mad dash for technological progress, instead slapping on bells and whistles that boost performance with little regard for biological plausibility.

Demonstrating polymer entanglement

From Steve Spangler I learnt this "party trick" demonstration of how the polymer molecules (polyethylene) in a plastic bag are entangled with one another. 

I was not sure that it would work as easily as it did for him. But it did!

Condensed Matter Physics: A Very Short Introduction out now!

Hard copies of my book can now be purchased directly from Oxford University Press. 

After a long wait and a lot of work, it was great to finally see it in print. I am very happy with the quality of the typesetting and the figures.

I look forward to getting feedback from readers.