On (De-)Centralized Communications: Part 1

#Introduction

At IETF121 in Dublin earlier this year, I was surprised to see my name appear on a number of slides, starting with the Discussion of Next Steps presentation Dirk and Lixia gave. I’d been commenting on the list on terminology related to the problem of Internet centralization.

The topic is fairly widely discussed these days. Other than DINRG, which of course has been formed to collect research on the issue, recent years also saw the publication of Mark Nottingham’s RFC on Centralization, Decentralization, and Internet Standards¹ on the topic. I remember making a few comments on it before it was finalized.

This year’s DINRG also saw mention of Mark McFadden’s two current drafts on much the same topic, A Taxonomy of Internet Consolidation² and On the Effects of Internet Consolidation³. These texts all wrestle with the problem of defining what “centralization” actually means, because different people understand different things when they refer to that term.

Finally, Christine Lemmer-Webber posted a long, multi-day thread on this topic over on fedi which I am not going to link to, because her blog post “Re: Re: Bluesky and Decentralization” captures the whole thing better than a fedi thread can.

I find myself getting embroiled in the same discussions on definitions, when for me, Baran’s Introduction to Distributed Communications Networks⁴ still does a great job.

But, this time around, something suddenly clicked, and I can now talk better about why I connect so well with his terms when not everybody does. So let’s examine how his descriptions hold up today.

#Baran

As a quick recap, note that Baran was writing at a time before the Internet, when he was trying to convince the military that lack of centralization was good for communications. In this, he was mostly concerned with resilience, specifically in the face of loss of a communications node.

The three models he compared were:

Centralized

In this model all nodes communicate via a central node.

Decentralized

Here, many nodes connect to fewer nodes, which in turn connect to each other.

Distributed

In the last model, all nodes connect to two or more other nodes, providing resilience against the loss of one of their connections.

Christine argued that the “decentralized” model here is just a variation of the “centralized” one, and that is (mostly) correct. To quote Baran directly on this:

For example, type (b) in Fig. 1 shows the hierarchical structure of a set of stars connected in the form of a larger star with an additional link forming a loop.

Where I disagree a little with her assessment is that Baran also writes about a “mixture of star and mesh” components; arguably, the diagram does not do this mixture justice.

That actually is important, even though it may seem a little bit of a nit to pick.

#The Nature of the Internet

For this article, I actually went through the effort of rebuilding the “distributed” model as an SVG, so I can extend it for illustration purposes. Looks nice, right?

I didn’t do this for the fun of it, however. The reason I did this was so I could play around with it, extend it a little, and then colour in some nodes.

Because I think part of the reason this discussion is so difficult, is that people not only differ in what they mean e.g. by these terms, but also because the diagrams do not actually relate well to the shape of the Internet today.

In fact, other discussions on the Internet’s shape use completely different terminology, and there is – obviously one might say – overlap in meaning that is obscured by using those different words.

Now before going into more detail here, let’s just quickly add a few more nodes to the diagram, yes?

All I did was add a bunch more nodes on the outside of the previous mesh, and connect them to the edge nodes of that mesh. Perhaps you might know where I’m going with this?

We often throw around terms like “the edge” and “core Internet”, and sometimes discuss the “end-to-end” principle. But I think it’s worthwhile putting those terms together in context with Baran’s terminology.

And in case you’ve been paying attention and are thinking that this looks a lot like a more complex variant of Baran’s “decentralized” model, well done! But this extended diagram also allows us to illustrate some points a little better.

First things first, let’s give those new nodes a little bit of colour. Those are the end nodes, like our mobile devices or laptops that we connect to the Internet at large.

Now the relation to Baran’s original mesh diagram is more visible. Nice.

So these end nodes all connect to an edge node. That’s pretty much the definition of the edge, although some might throw both kinds of nodes together into the definition of the latter.

Often enough you’ll also encounter the term “edge router” to differentiate nodes on the far end, and nodes they’re connecting to.

I think it’s very useful to keep them apart more clearly, however, precisely because only then can we discuss their distinction! So we’ll use end node and edge node for the router.

We can give those edge nodes a colour as well!

In this diagram, this edge is one layer of nodes deep. I think that’s pretty fair, but it’s necessary to point out that this is a simplification of sorts. Where I live, my home router connects to a box that lives on the street corner in some cabinet, just like the neighbour’s home routers.

The box in the cabinet then connects to some box in the next larger town over.

I know this, unfortunately, because a few times in the last few years, I couldn’t get an Internet connection at home. In every case, it turned out that the connection from the street corner to that next town was lost.

I think a fair definition of an edge node is any single node that is in the direct line of connectivity between the end node and the core nodes.

Core nodes. They also need a colour.

The core nodes then form the, well, “core Internet”, i.e. that fabric of data centers and carriers most of us don’t typically think about.

As a quick aside, this is relatively similar to the terminology one might use in maintaining your own network infrastructure. The “core” are routers and switches without external connection. The “edge” would perhaps consist of the “provider edge” and the “customer edge”, two routers that together connect your “core” to whatever the customer maintains.

If you use MPLS (label switching), it would be the job of the “label edge router” to perform the translation from the IP-based routing outside, and the label switching in the “core”.

If you paid extra close attention, you’ll see that the edge nodes here may also have direct connectivity between each other. That’s not always going to be the case! Whether the edge is my home router or some IoT router that’s providing connectivity for a bunch of sensors, chances are good it only has one uplink.

What happens if we remove that?

Oops. Now we have a problem. In three of the four corners of the graph, we now have edge nodes that do not have any connectivity to the core! Could this diagram be wrong?

No, it just illustrates that the notion that there may be a “provider edge” and a “customer edge” extends to the overall Internet as well, even if “provider” and “customer” are not quite the right terms.

We might instead call them outer edge nodes and inner edge nodes, where the main distinguishing factor is that only at the “inner edge” will there be connectors to core nodes.

But instead of focusing on different types of nodes, in this last version of the diagram I instead coloured the outer edge-to-edge connectors differently. I think this better reflects the roles the nodes play; they’re all edge nodes, but their connectivity may differ.

Let’s summarize this:

1. The Internet does not have that much of a different (abstract) “shape” than individual networks, at least in terms of the roles of nodes and how they connect to the rest of the network.
2. End nodes exist at the outmost part of networks, and are connected to the core nodes via edge nodes.
3. Edge nodes may come in variants which we can loosely classify:
   • Outer edge nodes only connect to end nodes and other edge nodes.
   • Inner edge nodes also connect to core nodes.
4. Core nodes provide the routing and connectivity in the network that ultimately serve end nodes.

#Baran (Again)

With this examination of node types and their connectors now established, we can relatively quickly determine if Baran’s terminology of “decentralized” or “distributed” networks applies to the Internet.

The answer is clearly ambiguous:

1. When considering all kinds of nodes, we can see that the connectivity between end and edge nodes is more reminiscent of a decentralized network. While not all end nodes rely on the same edge nodes, the loss of a single edge node will cause a number of end nodes to lose the capability to reach anyone else.
   • If edge-to-edge links exist, this may be less of a severe issue, but actual network designs do not usually allow for this. It should not be discounted as a possibility for engineering more resilient networks.
2. Conversely, core nodes resemble the fully distributed network that Baran describes as the most resilient kind of network.

It is therefore both true to call the Internet a decentralized as well as a distributed network. The only difference is in whether you are more focused on end nodes or on the design of core and (inner) edge networks.

The coloured diagram is also useful as a visual representation of each node’s degree of centralization.

But that is something I’ll get into in the next post.