netlab status: show memory usage per node and per lab - #3794
Conversation
|
I contributed a PR to Containerlab over the weekend: srl-labs/containerlab#3325 These cgroups allow you to collect lots of stats directly: I suspect Linux sys/fs could give us all those numbers without going through those separate tools |
Just to make sure I understand: when your PR gets into a containerlab release, and when we switch to that release, and (optionally) when we update the clab.yml, we will be able to read these things directly. Right? |
Well, actually we could read it today:
The cgroup parent grouping can be helpful to get the aggregate, and makes the lookup slightly simpler. |
|
Aaah, I forgot to remove my Multilab commit... I'll remove it later. I didn't know about |
487513d to
b3b44a7
Compare
|
We could also use the ╰─❯ cat /run/libvirt/qemu/ml-2_aruba0.pid -p
1997096
╰─❯ cat /proc/1997096/cgroup -p
0::/machine.slice/machine-qemu\x2d2\x2dml\x2d2aruba0.scope/libvirt/emulator
╰─❯ cat /sys/fs/cgroup/machine.slice/machine-qemu\\x2d2\\x2dml\\x2d2aruba0.scope/memory.current -p
1977995264 |
Yeah, I found that recipe, and figured out it does not work on my Docker/Ubuntu combo 🤷♂️ @sdargoeuves -- I would suggest we use the orchestration tools (Docker/libvirt) to fetch the stats. Let them deal with the underlying OS mess. See also: https://www.youtube.com/watch?v=oebqlzblfyo |
|
Finally got a chance to start my main lab to look at the memory usage with this branch. The Now I can see which node(s) are the hungry ones... no surprises, but still nice to see! Here is an extract: |
I won't push the idea of the |
On my simple containers-only lab, it took 2.3 seconds instead of 0.3 seconds, so it looks like the 2-second add-on is pretty much constant? Should we add a '--memory' flag to trigger the memory usage collection, or just make it a default and accept the 2-second delay? |
Now it works for me as well (after I ran your code which ran docker stats)... or maybe I was just brain-dead. Anyway, if we use docker stats and virsh domstats, and things don't work as expected, it's someone else's problem. If we start using /sys/fs, it becomes ours, and it could turn into a whack-a-mole situation. |
|
On my main lab, that command is so slow, adding 2min is not an issue. But yes, adding a --memory option would be a nice way of making it optional |
ipspace
left a comment
ipspace
left a comment
There was a problem hiding this comment.
The code looks good to me, the only thing left is the question of whether we accept the 2-second delay of docker stats, or make this optional, or add a "collecting memory usage, please wait" line to the interactive printout (I can easily add one or the other; just let me know what you prefer).
My two cents: everything should run as fast as possible. It's 2026. Memory consumption is largely irrelevant in day to day operations for a user. You'll know when you exceed available memory: OOM killer will ruin your lab. Due to the nature of the beast, it's also something you can do nothing about. --top or --memory is ideal. |
|
If you have no other containers running: systemd-cgtop -m -b -n 1 -r /system.slice | grep -E '^[[:space:]]*/system.slice/docker-[a-f0-9]+.scope[[:space:]]' | awk '{sum+=$4} END {print sum/1024/1024 " MB total"}' |
|
@sdargoeuves -- I decided to merge this and then add bells and whistles. Thanks again, a wonderful addition ;) |
4 participants
As it has been discussed on the Network Automagic podcast, I wanted to give (AI a) go at showing the memory usage of a lab, per node.
This is the output for a lab with libvirt+clab:
Disclosure: high risk of AI slop from this point.
Implementation — each provider's
get_lab_statusenriches its status box with amemorykey;fetch_node_statuscopies it into the node status and sums the total intolab_state.memory(so it also lands in--format json/yaml).docker stats --no-stream --format json→MemUsage, "used" half only. The value after the slash is the host total when no limit is set (always, for netlab-generated labs), so it's dropped.virsh domstats --balloon --list-running→balloon.rss(host RSS of the QEMU process). Falls back toballoon.currentsuffixed(max)when the guest has no balloon driver. Domain names are matched back to Vagrant machine names via the_<node>suffix; no new keys are added to the status box.format_memory_size/parse_memory_sizehelpers, docker's1.521GiB/184.5MiBstyle.Both collectors are non-fatal — failures are
print_verboseonly, memory is extra info.Caveats / things you may want changed:
netlab statusfor a clab lab (docker stats --no-streamsamples every container on the host, not just the lab's).used / allocateddisplay, which could be useful for libvirt.