Merging topology fragments

[a-v-popov]

Is there a way to build topology from multiple fragments?
E.g. one file describes physical topology and another logical structure, so that the logical structure can be an overlay on physical topology which can be executed by different providers as needed.

[ipspace]

I had similar ideas when building BGP labs ;) Short answer: not at the moment (but you know things can happen ;)

The details:

• The _include directive can include another file (or a set of files) as a dictionary key, not at the top level, but that should be easy to fix. * However, _include merges dictionaries, and things that are lists will be overwritten, links being the prime example. However, if you want to write links as a dictionary (which is also an option), it should be possible.

What else have I missed?

[a-v-popov]

The _include directive can include another file

I was thinking more in terms of overlay topology rather namespace loader.

to write links as a dictionary (which is also an option)

What would be a dictionary value? A list of interfaces, which would be replaced on collision? Does it allow a single interface?

What else have I missed?

Maybe a stable merge key for links and binding for ports. E.g. something like

#topology.yml
links:
  - id: r1-r2-core
    name: primary-core
    interfaces: 
      - node: r1
        port: core
      - node: r2
        port: core

#physical.yml
nodes:
    r1:
        ports.core.ifname: GigabitEthernet0/0/3
        mgmt.ipv4: 10.0.0.101
    r2:
        ports.core.ifname: Ethernet7
        mgmt.ipv4: 10.0.0.102
        
links:
  - id: r1-r2-core
    mtu: 9000

and then either netlab up ... topology.yml for virtual environment or netlab (create|initial) ... topology.yml --overlay physical.yml for a physical one.

Or maybe an explicit port binding which can be selected by CLI:

bindings:
  physical:
    r1.core.ifname: GigabitEthernet0/0/3
    r2.core.ifname: Ethernet7

and then netlab ... --bindings physical topology.yml --overlay fancy.yml

Binding could also be a merge on an explicit interface id, but since interfaces seems to be internal data structure, it might be better be presented as a separate feature.

I am not sure about all of this though, more like investigating possibilities.

[jbemmel]

and then either netlab up ... topology.yml for virtual environment or netlab (create|initial) ... topology.yml --overlay physical.yml for a physical one.

We could enable netlab to accept multiple topology files, appending sections in the order they are listed. So in your case, the links from physical.yml would get appended to the links: section in topology.yml

No need for special --overlay flag - just multiple files merged in order of appearance

[jbemmel]

[jbemmel]

Something like this: #3592

[a-v-popov]

accept multiple topology files

Let's assume I am using some_netlab_cmd to start a topology. What would be the new command to run the same topology augmented with another fragment. With an explicit CLI switch that would be some_netlab_cmd --overlay augment.yml. Try to answer that question without one. But I don't think CLI is the problem here. Plus I see no reason why the two options would be mutually exclusive.

So in your case...

My case, or at least the one I had in mind, is to switch back and forth between virtual and physical setup, because those super fast magic containers can only get you so far without proper data plane. This seems to required a genuine merge, either explicitly via unique ids, or through bindings. At the moment I lean towards the latter, because I would prefer to express logical setup through abstract references and then bind them to actual identifiers like ifname or ifindex, if required. Although I think a general merging mechanism based on unique, potentially autogenerated, keys might be useful in other cases when one want to decompose information on about the same link, for example.

[a-v-popov]

I am not sure if we have the same mental model of the merge.

What I was thinking about is adding configuration from this post https://blog.ipspace.net/2022/05/netsim-hardware-lab/ on top of a topology tried and tested in a virtual environment. I personally feel that exact ifnames might be better off in a dedicated stanza rather than on links, but if there is never a need to name them twice that would be an implementation choice. But I am not very familiar with netlab nerd-DSL to be sure. Anyway, I think what is required for my idea to work is a procedure to merge the same object from different configurations, I don't think it exists.

According to the blog post you referenced "The values of the links dictionary can be lists of links or further dictionaries." So we end up with two lists of links anyway. One can then stipulate that if there is only one link in both of the lists then two lists are merged by combining the two links into a single one (using a procedure which is not defined yet). What effectively happens here is you treat full group path as a link id. So I would be much more in favor of id being explicit. Like I said you can autogenerate them as well in some cases. So there is an interesting question if you want to merge original grouped links stanza, effectively treating group path as prepend to an explicit link ID, or merge them after links are flattened, allowing to merge link objects from different groups. I think the latter is more interesting.

So the first question might be "What do we merge and how do we combine them?". Can we start from root of the typologies and merge them recursively combining the "objects" with matching ids?

I am not sure about "How do we specify what needs to be merged?". My original though was in terms of CLI, but if I need some non-trivial configuration to specify merge process, or provider change, maybe some sort of additional control file would be warranted.

Please let me know if I am missing something or misunderstood your comment.

[ipspace]

I am not sure if we have the same mental model of the merge.

I think we're pretty much in sync, and I agree with you that merging link data based on their IDs makes the most sense. Will open an enhancement issue first for link.id, then for the merging topologies part.

Now, let me ask you a somewhat unrelated question: is your primary use case having interface names for physical hardware? In that case, how about we add external.ifname interface attribute?

[jbemmel]

If we identify links by the nodes connected to them (e.g. links[a,b] for the first link between a and b) and assume link mapping happens in declaration order (to handle multi-link cases) then we wouldn't need separate identifying attributes. Links map because they connect the same set of nodes

[ipspace]

If we identify links by the nodes connected to them (e.g. links[a,b] for the first link between a and b) and assume link mapping happens in declaration order (to handle multi-link cases) then we wouldn't need separate identifying attributes. Links map because they connect the same set of nodes

In theory, that would work. In practice, I'm afraid to think about all the ways this could go wrong (and the fun someone would have troubleshooting it).

[a-v-popov]

how about we add external.ifname interface attribute

You meant per-provider ifname? In some sense it would be similar to the "binding" suggestion, but activated by choice of a provider, instead of a dedicated switch. It might be interesting, however, I would still prefer it to be specified in a separate fragment. Physical typologies are expensive and I expect them to be transient. The idea is to have virtual netlab topology as a primary base which could be used to establish high level validity of the design and produce lab artifacts to build a somewhat temporary physical lab. It might take several virtual-to-physical iterations and each iteration might use different physical equipment.

If we identify...

I think we agree that we need to identify objects to be combined. The easiest and most reliable way to do so would be to use some sort of an id field. AFIAK, there is no such an id for links. It might exist for nodes, but even there I think an explicit one won't hurt much. Once procedure to merge using those explicit ids is established it should be relatively straightforward to add some magic to generate those if needed.