Repository navigation
Provide put & get message passing primitives for the bsp1d backend #405
Description
Provide an interface for point-to-point communication between processes run by the bsp1d backend
{{ message }}
Provide an interface for point-to-point communication between processes run by the bsp1d backend
Activity
GiovaGa commentedon Dec 5, 2025
More precisely this feature will provide the following functions, inside the namespace
grb::spmd(?) :put(...),get(...)to read/write messages to/from another process, with specializations forgrb::Vector;grb::collectives::register_global(...),register_local(...)to register memory to be used for inter-process one-way memory access;@anyzelman am I missing something important?
anyzelman commentedon Dec 5, 2025
I suggest to create a new namespace instead of collectives, called
grb::rdma<>. The put, get, register_{global,local} should go in that namespace. For the third point, in your use case, are you sending ALP objects or non-ALP objects? (Are you always communicating reference vectors, cf. #403, for example?)If the latter (i.e., it's really memory that we expose to users), the roadmap looks good but if it's the former (which seems more C++-y to me), then whether an ALP object had been registered for RDMA communication should be part of the internal ALP state.
anyzelman commentedon Dec 5, 2025
Note btw (but I think you already noticed) that the related functionality is actually already implemented, so it's really all about the interface right?
(most of the implementation is here: https://github.com/Algebraic-Programming/ALP/blob/develop/include/graphblas/bsp1d/init.hpp#L50 )
GiovaGa commentedon Dec 5, 2025
I will be sending both ALP objects and scalars. In any case the RDMA interface for ALP objects will use the same functions as scalars, so we might as well expose it
GiovaGa commentedon Dec 5, 2025
Seemingly there is something, but it seems very little
anyzelman commentedon Dec 5, 2025
In this case I suggest that
put,get,register_{local,global}are strongly typed functions that take only the arguments that you need at the moment -- e.g.,grb::Vector< T, reference >and POD types (i.e., scalars). Then ALP internally can worry about translating it to registrations and RDMA requests of the right raw memory areas.anyzelman commentedon Dec 5, 2025
All BSP1D functionality goes back to the structures and functions there, although in the majority of cases we revert to collectives rather than direct RDMA. Still, for LPF-based collectives, the structs defined here are used, and also memory registration I believe works through the code there.
Also worth noting is that the code linked to is not a simple LPF wrapper to C++, the difference is the following:
LPF is super minimal in that one needs to actively reserve enough memory slots and enough space for in-flight messages before registering memory and before issuing RDMA requests. The ALP code linked to instead turns this into active management: the reserved buffer sizes are recorded, and if a memory registration is requested while the related buffer size is insufficient, ALP will make sure the related buffers are resized. This also implies a warning for this issue though: the way the buffer management works needs to be ensured to be compatible with the new exposed RDMA interface. This is much harder to do than previously, because in principle, a user can issue any arbitrary communication pattern in any form of overlap.
I think we can just take the easy way out, at least for a first version: if the user requests a registration that violates an active buffer constraint, we just throw an exception. A user can catch that exception and if caught, ask ALP to increase its buffer (in essence, thus, going back to LPF's semantics)
anyzelman commentedon Dec 5, 2025
This one is tricky btw -- LPF has functionality for it, the so-called synchronisation attributes which are designed to allow "weakening" synchronisation semantics while keeping the code structured. The weakest is the so-called zero-cost synchronisation, but I have strong reservations about exposing that to the ALP user directly: it's powerful, but very easy to make mistakes with. Let's maybe first get your use case to work with global fences, and then look at the code together to see how we can make it both performant and simple to use?
GiovaGa commentedon Dec 11, 2025
Thinking about a way to make RDMA work in a user-friendly way, I had the following idea:
We could allow RDMA to work only between a specified set of "associated" memory regions: when the global registration is done. We store internally the pairing between the given address and its memory slot.
When an RDMA memory access is done, the user will just give the local address (or object) and the PID whose memory he wants to access. The lpf memory slot is then recovered internally and the RDMA done between the objects globally registered together and thus associated at that time.
@anyzelman let me know if this seems a good way to proceed.
anyzelman commentedon Dec 16, 2025
Yes that's a good idea. In discussion just now (first test by replacing
nonblockingin the below withreference), the API could be as follows:grb::RC grb::rdma< BSP1D >::register( const grb::Vector< T, nonblocking > &vector )grb::RC grb::rdma< BSP1D >::register( const T &scalar)grb::RC grb::rdma< BSP1D >::localRegisterSize( const size_t )grb::RC grb::rdma< BSP1D >::put( const grb::Vector< T, nonblocking > &src, const size_t dst_pid, grb::Vector< T, nonblocking > &dst )grb::RC grb::rdma< BSP1D >::put( const T &src, const size_t dst_pid, T &dst )grb::RC grb::rdma< BSP1D >::get( const size_t src_pid, const grb::Vector< T, nonblocking > &src, grb::Vector< T, nonblocking > &dst )grb::RC grb::rdma< BSP1D >::get( const size_t src_pid, const T &src, T &dst )grb::RC grb::rdma< BSP1D >::fence( ... )Offsets, matrices, etc. we will not put in this MR, as the use case does not require it.
GiovaGa commentedon Dec 16, 2025
Small note: register is a C keyword, so I will use register_global