[Bug] Cannot bind multiple memory buffers to a single `NvlsConnection`

[liangyuRain]

Hi, we observe a strange behavior with NvlsConnection. When we bind two memory buffers to the same NvlsConnection, the DeviceMulticastPointer returned for the second buffer actually points to the first buffer. In other words, when binding multiple buffers to a singleNvlsConnection, the DeviceMulticastPointer returned always points to the first buffer that was bound.

We think the problem is at the following line:

mscclpp/src/nvls.cc

Line 212 in 2b9b18d

MSCCLPP_CUTHROW(cuMemMap((CUdeviceptr)(mcPtr), devBuffSize, 0, mcHandle_, 0));

The cuMemMap always maps mcPtr to offset 0 of mcHandle_, which is the offset of the first bound memory buffer, rather than the newly allocated offset above. However, cuda documentation (link) requires the offset to be 0 right now. I guess it is technically impossible to reuse mcHandle (and thus NvlsConnection) for multiple memory buffers. Do you think this is correct?

[liangyuRain]

We also observed a memory leak:

from mpi4py import MPI
from mscclpp import TcpBootstrap, Communicator, connect_nvls_collective
from mscclpp._mscclpp import RawGpuBuffer
import cupy as cp

rank = MPI.COMM_WORLD.Get_rank()
nranks = MPI.COMM_WORLD.Get_size()
print(f"{rank=}, {nranks=}", flush=True)

cp.cuda.Device(rank).use()
bootstrap = TcpBootstrap.create(rank, nranks)
uniq_id = None
if rank == 0:
    uniq_id = bootstrap.create_unique_id()
uniq_id_global = MPI.COMM_WORLD.bcast(uniq_id, 0)
bootstrap.initialize(uniq_id_global)
comm = Communicator(bootstrap)
for i in range(200):
    if i % 10 == 0 and rank == 0:
        print(f"{i=}", flush=True)
    bootstrap.barrier()
    mem = RawGpuBuffer(2 ** 30)
    conn = connect_nvls_collective(comm, list(range(nranks)), mem.bytes())
    handle = conn.bind_allocated_memory(mem.data(), mem.bytes())
    del handle
    del conn
    del mem
del comm, bootstrap

0: i=0                                              
0: i=10                                             
0: i=20                                             
0: i=30                                             
0: i=40                                             
0: i=50                                                                                                 
0: i=60                                                                                                 
0: i=70                                                                                                 
0: i=80                                             
0: i=90                                                                                                 
0: i=100                                            
0: i=110                                                                                                
0: i=120                                                                                                
0: Traceback (most recent call last):                                                                   
0:   File "/.../debug.py", line 24, in <module>                            
0:     handle = conn.bind_allocated_memory(mem.data(), mem.bytes())                                     
0: mscclpp._mscclpp.CuError: (2, 'Call to cuMulticastBindAddr(mcHandle_, offset , devicePtr, devBuffSize
, 0) failed./.../mscclpp/src/nvls.cc:208 (Cu failure: out of memory)')

[liangyuRain]

The memory leak is solved by adding cuMemRelease(mcHandle_) to NvlsConnection::Impl::~Impl().