mi: Introduce asynchronous event message handling

[chorkin]

Added new functionality to mi.c and mi-mctp.c to handle AEMs. Included new example mi-mctp-ae.c for usage.

[chorkin]

@jk-ozlabs, there's something funky going on with the existing mi-mctp tests. I was hoping you might shed some light here. Also looking for guidance on how we can add new tests to handle these changes.

[chorkin]

@igaw, it seems we have build checks that have been added since the last time I upstreamed something. But they're all failing for me here. Any guidance you can provide?

[jk-ozlabs]

Neat, I'll take a look!

[jk-ozlabs]

I think the overall concept here is sound, but we are missing a few implementation details for an overall feasibility check too. There are some style things, but we can address those once the general framework is settled. I have added some comments in a review.

I've been thinking about how we should be handling the multiple-endpoint cases, as we can only have one socket listening on incoming NVMe-MI message types. A couple of options there:

• a shared socket for all AEM events on all endpoints (eg, on the nvme_mi_root_t), where a call to fetch AEM events on any enpoint will receive all events, and queue those that are not for the current. A subsequent call for any other endpoints would first check that queue before receiving more directly from the socket

or:

• we add socket syscall semantics for AF_MCTP, to specify the remote MCTP EID on the listening socket, and stick with one per nvme_mi_ep_t. this would require kernel changes, but I think that's probably a common use-case that would be good to support.

I figure the latter is probably best, so will take a look at what's required for that.

[jk-ozlabs]

(BTW, I would suggest converting to a draft while still working out details)

[igaw]

@igaw, it seems we have build checks that have been added since the last time I upstreamed something. But they're all failing for me here. Any guidance you can provide?

The build container cleanup task eagerly removed containers.

[chorkin]

Collaborator

I think the overall concept here is sound, but we are missing a few implementation details for an overall feasibility check too. There are some style things, but we can address those once the general framework is settled. I have added some comments in a review.

I've been thinking about how we should be handling the multiple-endpoint cases, as we can only have one socket listening on incoming NVMe-MI message types. A couple of options there:

• a shared socket for all AEM events on all endpoints (eg, on the nvme_mi_root_t), where a call to fetch AEM events on any enpoint will receive all events, and queue those that are not for the current. A subsequent call for any other endpoints would first check that queue before receiving more directly from the socket

or:

• we add socket syscall semantics for AF_MCTP, to specify the remote MCTP EID on the listening socket, and stick with one per nvme_mi_ep_t. this would require kernel changes, but I think that's probably a common use-case that would be good to support.

I figure the latter is probably best, so will take a look at what's required for that.

@jk-ozlabs , I wasn't aware of this limitation. I think the kernel changes are probably the best solution here, but the case we are talking about is probably not going to be a primary use case. I was wondering if we can come up with a good strategy here to solve this multi-endpoint AEM issue without holding up the initial PR. Some thought's I had were:

1. Provide this as a known limitation for the current PR and work on kernel changes in background. Add a follow up to remove this limitation.
2. Implement the shared socket for a short term solution in this PR so there will be no limitation. Follow up with kernel changes later to simplify the code and memory usage of the application. I'll see if I can prototype this if I have time.

[chorkin]

@igaw , I've addressed most of the checkpatch issues, but I can't resolve the packed ones. When I add __packed the structures aren't actually getting packed correctly. I also see plenty of usage in the codebase in the style I'm using.

I also see kdoc failure. What's the best way for me to dig into what's going on here? Is there a way to build the documentation locally?

@jk-ozlabs , I believe the mi-mctp tests are failing due to issue I flagged for you earlier.

[chorkin]

Sounds like we're going to be enabled to build kernel changes so I will test this

super!

Do I just need to copy your example code as is to do it or are there other things I'll need to do?

You'll just need to:

1. incorporate that mctp-peer-bind branch in to your kernel tree.
2. Add the diff I included above to libnvme

for (1), easiest approach (if you already have a custom kernel) would be to grab commit CodeConstruct/linux@2e9c0d5 and onwards, and apply those to your custom kernel. Or, just build that mctp-peer-bind branch directly if you have no other changes that you need

for (2) feel free to incorporate that into your change, or I can provide as an actual commit if you prefer.

Alrighty! I was able to build our QEMU today and test with your commits on top. I did two tests:

#1) Without your changes, ran with the new peer-bind code in libnvme-mi, and verified an error is returned in the example app since this is not supported yet
#2) With your changes, did the same test. Verified no error and verified I can trigger AEs which show up in my example app.

What I'm not able to test is multiple endpoints with AEs because that's just not something this QEMU setup will allow at this point, but if you are confident in your kernel changes, or can test that bit on your end, I think we're good to go!

Cheers and have a great weekend!

[jk-ozlabs]

nice one!

The multiple-endpoints functionality (at the MCTP level) is something we have tests for, so I'm fine to cover that on our side.

[We have a facility for multiple NVMe-MI-device emulation under qemu, but the device-side doesn't support AEs at present. I'll give this a go once that is implemented though]

[chorkin]

@jk-ozlabs are we good to complete this, then?

[jk-ozlabs]

Looking good!

I have done a bit of a nit-picky review here; most is not super important.

However: we will want to make sure we have the mi.h changes correct, as they'll become very difficult to modify once merged. If nothing else take a look through those and let me know if you disagree on any points there.

Thanks for all the work on this!

[jk-ozlabs]

The API all looks good to me!

[chorkin]

@igaw , this is ready to merge.

[igaw]

Oh, it's a non trivial rebase. @chorkin could you rebase it against the current master, I think I'll break a thing or two if I do it myself. Sorry.

[chorkin]

I need to do one final check, @igaw . Will post here when done.

[chorkin]

I need to do one final check, @igaw . Will post here when done.

@igaw, rebased and ready to merge.

[igaw]

just updated the map file, sorted the lines alphabetically.

[igaw]

Thanks!

[jk-ozlabs]

🥳

[chorkin]

Thanks everyone! This is such a cool new feature; I'm glad we could get it in!