You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I love that this library is sans-io, but it makes testing a bit tricky in that our reference implementations (i.e. the Linux Kernel and iproute2) don't play along.
Ideally, everything implementing the Nla trait would get its own set of tests, but this is very tricky with the current testing strategy of collecting reference messages.
We could break each message down into individual Nlas and test them separately, but this would require a large amount of manual work and is likely to be error-prone.
As things stand, we have larger tests which examine entire messages.
These tests are very helpful, but they have also required me to spend several hours with a debugger trying to hunt down the exact source of the error when they fail.
An example of a scope issue I have faced is in testing the c-vlan key of flower for DRAFT: tc flower support #111.
Matching on c-vlan requires matching on s-vlan, which requires matching on the ethertype.
Thus, any test I write for c-vlan with this method inevitably drags in functionality for the test which would ideally be out of scope for that test.
Limited coverage and a lot of manual work:
I have been trying to write tests for DRAFT: tc flower support #111 but I am struggling to get good quality test coverage.
Composing an adequate number of test cases is proving to be taxing with this method.
Test opacity:
It is challenging to look at the hex dumps for the messages and correlate them with test failures.
In particular, I am often uncertain if the error I am seeing is due to a bug in the library or a bug in the test; I simply don't always know exactly what the message is saying in the first place.
Accounting for endianness issues with this method is tricky:
If the messages are captured on a little-endian machine, then the tests will fail on big-endian machines and vice versa; the hard-coded messages are simply incorrect on the opposite endianness architecture.
Accounting for endianness is especially important in that endianness is highly inconsistent in the context of netlink: it often swaps between network-endian and native-endian within the space of a single message.
Doing a direct dump on a nlmon interface is often noisy.
Many people use tools like systemd-networkd, NetworkManager, or some other network management daemon.
These daemons, along with tools like docker and kubernetes can generate a lot of noise on the nlmon interface which I need to sort through to find the messages I am interested in.
This is both inconvenient and a potential waste of development time.
Are you really sure you got the message you wanted?
Ideas for improvement:
I have a few ideas for improving the testing strategy for this library.
Idea 1: (ab)use strace to capture netlink messages
This method doesn't address all the above concerns by any means, but it can make things a little easier in my experience.
I don't suggest that we use this script as-is (it is quite hacky).
The goal of the script is to (ab)use strace and standard linux tools into spitting out a simple yaml file containing all sent and received netlink messages for a given command.
In this case we see a "conversation" between the iproute2 and the kernel.
There are two netlink sockets open (file descriptors 3 and 4).
We can see both the raw hex of all messages and strace's interpretation of them.
Let me know if you like this concept.
If so, I will write up a proper version of it (i.e., not a quick bash script).
Likely I would use eBPF / Aya to do this properly.
This method could be integrated into the test suite at any level
As a manual process / recipe for developers to more easily compose tests.
As a utility script which we run on each supported architecture to generate data for the tests to consume.
As a part of the test suite itself.
I tend to think option 3 is a bad idea as it will inevitably lead to a lot of complexities and the hard requirement to run tests as root.
Things we could do with the yaml output:
Loop over the messages in our tests and assert that at least one of the messages parses into some expected structure.
Assert that no component of the conversation parsed as DefaultNla (i.e. that we actually can parse the whole conversation).
Add tests for necessary (if not sufficient) conditions
I think we can agree that every impl of Nla + Emittable should have the property that it must be able to be emitted and then parsed back to the same structure you started with.
If you have a mismatch between your emit and parse functions, you will likely catch it. Including
Complete failure to parse
Complete failure to emit
Inconsistencies (especially endianness) between parse and emit
Incorrect buffer length calculations (often, but not always)
More, these tests
are of very limited scope (you can write them on a per Nla basis),
are straightforward to write (to the point that they may be automatically generated via a proc-macro),
require no IO,
do not require any hard coded buffers,
and are fuzz-testing friendly.
I really like this method, and I am happy to write such tests, either by hand, or I can compose a proc macro we can use to generate such tests automatically.
Cooperation with pyroute2 or any of the go-netlink libraries
We are currently (and very reasonably) using iproute2 as a reference implementation, but there is no rule making iproute2 canonical.
If we can cooperate with the other libraries in this space we may be able to come up with a sort of "dictionary" of example netlink messages / nla segments which we could all use to test our libraries.
I am not certain about the feasibility of this idea, but I think it is worth considering.
Let me know what you think :)
The text was updated successfully, but these errors were encountered:
Testing strategy improvement ideas
I love that this library is sans-io, but it makes testing a bit tricky in that our reference implementations (i.e. the Linux Kernel and iproute2) don't play along.
The problems
The current testing strategy for this library relies on capturing netlink messages on nlmon interfaces.
While this is a fairly straightforward method, it has a few drawbacks:
Large testing scope:
Ideally, everything implementing the
Nlatrait would get its own set of tests, but this is very tricky with the current testing strategy of collecting reference messages.We could break each message down into individual
Nlas and test them separately, but this would require a large amount of manual work and is likely to be error-prone.As things stand, we have larger tests which examine entire messages.
These tests are very helpful, but they have also required me to spend several hours with a debugger trying to hunt down the exact source of the error when they fail.
An example of a scope issue I have faced is in testing the c-vlan key of flower for DRAFT: tc flower support #111.
Matching on c-vlan requires matching on s-vlan, which requires matching on the ethertype.
Thus, any test I write for c-vlan with this method inevitably drags in functionality for the test which would ideally be out of scope for that test.
Limited coverage and a lot of manual work:
I have been trying to write tests for DRAFT: tc flower support #111 but I am struggling to get good quality test coverage.
Composing an adequate number of test cases is proving to be taxing with this method.
Test opacity:
It is challenging to look at the hex dumps for the messages and correlate them with test failures.
In particular, I am often uncertain if the error I am seeing is due to a bug in the library or a bug in the test; I simply don't always know exactly what the message is saying in the first place.
Accounting for endianness issues with this method is tricky:
If the messages are captured on a little-endian machine, then the tests will fail on big-endian machines and vice versa; the hard-coded messages are simply incorrect on the opposite endianness architecture.
Accounting for endianness is especially important in that endianness is highly inconsistent in the context of netlink: it often swaps between network-endian and native-endian within the space of a single message.
Test failures on big-endian can be seen by running miri on their preferred big-endian target.
cargo +nightly miri test --target s390x-unknown-linux-gnuNoisy captures:
Doing a direct dump on a
nlmoninterface is often noisy.Many people use tools like systemd-networkd, NetworkManager, or some other network management daemon.
These daemons, along with tools like docker and kubernetes can generate a lot of noise on the
nlmoninterface which I need to sort through to find the messages I am interested in.This is both inconvenient and a potential waste of development time.
Are you really sure you got the message you wanted?
Ideas for improvement:
I have a few ideas for improving the testing strategy for this library.
Idea 1: (ab)use strace to capture netlink messages
This method doesn't address all the above concerns by any means, but it can make things a little easier in my experience.
I cooked up a quick and dirty proof-of-concept script to make strace do some of this work for us.
I don't suggest that we use this script as-is (it is quite hacky).
The goal of the script is to (ab)use strace and standard linux tools into spitting out a simple yaml file containing all sent and received netlink messages for a given command.
Here are the contents of
/tmp/trace.ymlafter running that script on my dev machine (basically, a netlink trace ofip route)In this case we see a "conversation" between the iproute2 and the kernel.
There are two netlink sockets open (file descriptors 3 and 4).
We can see both the raw hex of all messages and strace's interpretation of them.
Let me know if you like this concept.
If so, I will write up a proper version of it (i.e., not a quick bash script).
Likely I would use eBPF / Aya to do this properly.
This method could be integrated into the test suite at any level
I tend to think option 3 is a bad idea as it will inevitably lead to a lot of complexities and the hard requirement to run tests as root.
Things we could do with the
yamloutput:DefaultNla(i.e. that we actually can parse the whole conversation).Add tests for necessary (if not sufficient) conditions
I think we can agree that every
implofNla + Emittableshould have the property that it must be able to be emitted and then parsed back to the same structure you started with.For example, you can write tests like this
This property seems very powerful to me.
If you have a mismatch between your
emitandparsefunctions, you will likely catch it. IncludingMore, these tests
Nlabasis),I really like this method, and I am happy to write such tests, either by hand, or I can compose a proc macro we can use to generate such tests automatically.
Cooperation with pyroute2 or any of the go-netlink libraries
We are currently (and very reasonably) using iproute2 as a reference implementation, but there is no rule making iproute2 canonical.
If we can cooperate with the other libraries in this space we may be able to come up with a sort of "dictionary" of example netlink messages / nla segments which we could all use to test our libraries.
I am not certain about the feasibility of this idea, but I think it is worth considering.
Let me know what you think :)
The text was updated successfully, but these errors were encountered: