tests: replace mqtt-logger with custom mosquitto log plugin

[reubenmiller]

Proposed changes

Improve the MQTT broker observation and test assertions by replacing the mqtt-logger with a custom mosquitto plugin which logs all mqtt messages directly within mosquitto. It provides more information about the messages as it does not have to adhere to the mqtt protocol.

The "Should Have MQTT Messages" now uses a cli client which is custom written for parsing the log files produced by the new mosquitto plugin.

The code for the mosquitto plugin is located under thin-edge/mosquitto-log, and it requires mosquitto 2.1.x which is made available via a custom built/packaged version of mosquitto thin-edge/tedge-mosquitto.

Notable changes:

Since mosquitto 2.1.x is now used, it brings some additional most of which are detailed in the 2.1.0 release notes:

• password_file is now a plugin and not supported by default. The plugin (.so) file is included within the tedge-mosquitto-gnu packaging
• the password file now needs to be owned by the mosquitto process otherwise it can't be read
• per_listener_settings has been deprecated and instead there are listener specific settings, e.g. allow_anonymous => listener_allow_anonymous

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
• Documentation Update (if none of the other choices apply)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

Checklist

• I have read the CONTRIBUTING doc
• I have signed the CLA (in all commits with git commit -s. You can activate automatic signing by running just prepare-dev once)
• I ran just format as mentioned in CODING_GUIDELINES
• I used just check as mentioned in CODING_GUIDELINES
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Further comments

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %	⏱️ Duration
953	0	7	953	100	3h1m42.309243s

[Bravo555]

Overall good direction, but I'm curious if it could make some tests flaky? I ran the entire suite a few times and ended with 4-6 failed tests, "Entity Store Compaction" seems to be particularly flaky for me, failing when run individually, but it doesn't seem to rely on the changed MQTT message matching.

[Bravo555]

"Entity Store Compaction" seems to be particularly flaky for me

Looking at the test report, /etc/tedge/.agent/entity_store.jsonl has only 4 entities, one of them being the compaction_count test device, so the flakiness most likely comes from this command:

${count}=    Execute Command    grep -c "compaction_count" ${ENTITY_STORE}    strip=${True}

completing before the compaction process is completed. So the flakiness I'm experiencing is probably not related to the change.

[reubenmiller]

Overall good direction, but I'm curious if it could make some tests flaky? I ran the entire suite a few times and ended with 4-6 failed tests, "Entity Store Compaction" seems to be particularly flaky for me, failing when run individually, but it doesn't seem to rely on the changed MQTT message matching.

I wouldn't rule out mosquitto 2.1.x having some slightly different behaviour than previous versions, but I think in general a lot of our tests are by nature flaky where it seems to come and go over time. But obviously it is something that we have to continuously work on to improve.

Update (sorry I originally added this update to your comment): I'm working on a follow up PR which looks to address some flaky tests that I've seen locally, and I'll include this test that you've mentioned and replace the line with Execute Command cmd=test $(grep -c "compaction_count" ${ENTITY_STORE}) -lt 10 which be more flexible in waiting for the compaction to complete.

[albinsuresh]

LGTM.

[albinsuresh]

Update or remove these?

[reubenmiller]

Yeah I'll remove this section now since it is very dated.

[reubenmiller]

Removed in f29a88c