[Feature]: Nightly release with statically-linked and/or rocm-prefix'd libLLVM20

[wsmoses]

Suggestion Description

Many projects also use LLVM, causing issues if linkinag against/dlopeining comgr, which itself uses libLLVM20.

At least for my use cases, the fact XLA has its own LLVM hits an issue when the dlopen init for comgrs LLVM is initialized.

ideally LLVM is built with a relevant rocm suffix (like below)
# set a SONAME suffix for FreeBSD JuliaLang/julia#32462
CMAKE_FLAGS+=(-DLLVM_VERSION_SUFFIX:STRING="rocm")
# Aggressively symbol version (added in LLVM 13.0.1)
CMAKE_FLAGS+=(-DLLVM_SHLIB_SYMBOL_VERSION:STRING="ROCM_LLVM_${LLVM_MAJ_VER}.${LLVM_MIN_VER}")

Also nice would be statically linking in libLLVM into comgr

#include <dlfcn.h>
int main() {
  # XLA build built against theRock, specifically: https://github.com/ROCm/TheRock/releases/download/nightly-tarball/therock-dist-linux-gfx110X-dgpu-7.0.0rc20250714.tar.gz
  dlopen("lib/libReactantExtra.so", RTLD_LAZY | RTLD_DEEPBIND);
}

Program received signal SIGSEGV, Segmentation fault.                                                                                                                                                                  
0x00007fffbd4a8ae3 in _GLOBAL__sub_I_ELFObjectFile.cpp () from /mnt3/wmoses/git/Yggdrasil/R/Reactant/re2/lib/llvm/lib/libLLVM.so.20.0git                                                                              
(gdb) bt                                                                                                                                                                                                              
#0  0x00007fffbd4a8ae3 in _GLOBAL__sub_I_ELFObjectFile.cpp () from /mnt3/wmoses/git/Yggdrasil/R/Reactant/re2/lib/llvm/lib/libLLVM.so.20.0git                                                                          
#1  0x00007ffff7fca71f in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7fffffffe058, env=env@entry=0x7fffffffe068) at ./elf/dl-init.c:74                                                        
#2  0x00007ffff7fca824 in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at ./elf/dl-init.c:120                                                                       
#3  _dl_init (main_map=0x5555555592c0, argc=1, argv=0x7fffffffe058, env=0x7fffffffe068) at ./elf/dl-init.c:121                                                                                                        
#4  0x00007ffff7fc65b2 in __GI__dl_catch_exception (exception=exception@entry=0x0, operate=operate@entry=0x7ffff7fd1cc0 <call_dl_init>, args=args@entry=0x7fffffffda80) at ./elf/dl-catch.c:211                       
#5  0x00007ffff7fd1d7c in dl_open_worker (a=0x7fffffffdc30) at ./elf/dl-open.c:829                                                                                                                                    
#6  dl_open_worker (a=a@entry=0x7fffffffdc30) at ./elf/dl-open.c:792                                                                                                                                                  
#7  0x00007ffff7fc651c in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffdc10, operate=operate@entry=0x7ffff7fd1ce0 <dl_open_worker>, args=args@entry=0x7fffffffdc30) at ./elf/dl-catch.c:237          
#8  0x00007ffff7fd2164 in _dl_open (file=0x555555556004 "lib/libReactantExtra.so", mode=<optimized out>, caller_dlopen=0x555555555165 <main+28>, nsid=<optimized out>, argc=1, argv=0x7fffffffe058,                   
    env=0x7fffffffe068) at ./elf/dl-open.c:905                                                                                                                                                                        
#9  0x00007ffff7c981a4 in dlopen_doit (a=a@entry=0x7fffffffdee0) at ./dlfcn/dlopen.c:56                                                                                                                               
#10 0x00007ffff7fc651c in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffde20, operate=0x7ffff7c98140 <dlopen_doit>, args=0x7fffffffdee0) at ./elf/dl-catch.c:237                                      
#11 0x00007ffff7fc6669 in _dl_catch_error (objname=0x7fffffffde88, errstring=0x7fffffffde90, mallocedp=0x7fffffffde87, operate=<optimized out>, args=<optimized out>) at ./elf/dl-catch.c:256                         
#12 0x00007ffff7c97c83 in _dlerror_run (operate=operate@entry=0x7ffff7c98140 <dlopen_doit>, args=args@entry=0x7fffffffdee0) at ./dlfcn/dlerror.c:138                                                                  
#13 0x00007ffff7c9825f in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>) at ./dlfcn/dlopen.c:71                                                                        
#14 ___dlopen (file=<optimized out>, mode=<optimized out>) at ./dlfcn/dlopen.c:81                                                                                                                                     
#15 0x0000555555555165 in main ()                                                                                                                                                                                     
(gdb) disassemble                                                                                                   

cc @ftynse @giordano

Operating System

Ubuntu

GPU

No response

ROCm Component

Comgr

[stellaraccident]

This landed on Friday, and we will be flipping the flag this week when a dependent patch lands all tests pass: b1bdc0b.

While we will also be doing some other items like you suggest, they are not sufficient to truly make co-existence robust (I have quite a number of issue reports to draw from which show the various ways that symbol versioning alone is insufficient to avoid issues). We may implement a static linking option as well, but getting the dynamic case working reliably is a priority for us given the size benefits.

The only true thing for a system component like ROCm to do for robustness is to completely hide the fact that it is based on LLVM when used as part of the normal runtime libraries. There are really only two ways to do this which cannot demonstrably interfere with other libraries: a) statically link and hide all LLVM symbols in comgr (if any symbols are left in the global namespace, than many improperly linked LLVM dependent libraries may prefer the ROCm LLVM symbols regardless of versioning, etc), or b) do what the above patch does and only depend on comgr/LLVM via a delay loaded trampoline in a different linker namespace (note that the core runtime was already delay-loading it with dlopen but there are other direct dependencies across ROCm on libamd_comgr.so which were not, so the above patch centralizes the delay-loading of the compiler and puts it behind dlmopen so that it resides in its own namespace and cannot create conflicts in the global namespace).

[wsmoses]

Stella you are a hero as always, let me know when there's a tarball release I can use!

[harakas]

I found this issue and believe I have something related to report:

I'm trying to build migraphx on top of TheRock and I get this kind of error:

$ /opt/rocm/bin/migraphx-driver compile model.onnx --gpu --enable-offload-copy  -o out.mxr
Running [ MIGraphX Version: 2.13.0.20250912-42-0-g524839ac9-dirty ]: /opt/rocm/bin/migraphx-driver compile model.onnx --gpu --enable-offload-copy -o model.mxr
[2025-10-01 17:31:57]
Compiling ... 
Reading: model.onnx
LLVM ERROR: Option 'default' already exists!
Aborted (core dumped)

"Option 'default' already exists!" points to multiple LLVM-s battling it out. I did compile with -DTHEROCK_FLAG_COMGR_DELAY_LOAD:

/opt/rocm/lib/migraphx/lib$ ldd libmigraphx_gpu.so|grep comgr
	libamd_comgr_loader.so.1 => /opt/rocm-therock.3/lib/migraphx/lib/./../../libamd_comgr_loader.so.1 (0x00007fb74ea00000)

On the other hand.

I modified TheRock to build a static LLVM instead and then migraphx works (after fixing another bug inside it related to PIE/ASLR).

So why is it not working with THEROCK_FLAG_COMGR_DELAY_LOAD or is this not related @stellaraccident?

[harakas]

Gdb:

(gdb) break dlopen
Breakpoint 1 at 0x7fffeba00fe0: file dlopen.c, line 75.
(gdb) break llvm::cl::Option::addArgument
Breakpoint 2 at 0x7ffe9639c170 (4 locations)
(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /opt/rocm-therock.3/bin/migraphx-driver compile model.onnx --gpu --enable-offload-copy -o out.mxr
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 2, 0x00007ffff684ae30 in llvm::cl::Option::addArgument()@plt () from /opt/rocm-therock.3/bin/../lib/migraphx/lib/libmigraphx_gpu.so.2013000
(gdb) c 2000
Will ignore next 1999 crossings of breakpoint 2.  Continuing.

Breakpoint 2, 0x00007ffff684ae30 in llvm::cl::Option::addArgument()@plt () from /opt/rocm-therock.3/bin/../lib/migraphx/lib/libmigraphx_gpu.so.2013000
(gdb) c 2000
Will ignore next 1999 crossings of breakpoint 2.  Continuing.
Running [ MIGraphX Version: 2.13.0.20250912-42-0-g524839ac9-dirty ]: /opt/rocm-therock.3/bin/migraphx-driver compile model.onnx --gpu --enable-offload-copy -o out.mxr
[2025-10-01 18:44:41]
Compiling ...
Reading: model.onnx
[New Thread 0x7fffa93ff700 (LWP 7683)]
[New Thread 0x7ffea89ff700 (LWP 7684)]
[Thread 0x7ffea89ff700 (LWP 7684) exited]

Thread 1 "migraphx-driver" hit Breakpoint 1, __dlopen (file=0x7fffba4ad280 "libamd_comgr.so.3", mode=2) at dlopen.c:75
75      dlopen.c: No such file or directory.
(gdb) bt 3
#0  __dlopen (file=0x7fffba4ad280 "libamd_comgr.so.3", mode=2) at dlopen.c:75
#1  0x00007fffb0c0130a in amd_comgr_stub_dlopen () from /opt/rocm-therock.3/bin/../lib/libamd_comgr_loader.so.1
#2  0x00007fffba48ebe4 in ?? () from /opt/rocm-therock.3/bin/../lib/libamdhip64.so.7
(More stack frames follow...)
(gdb) c
Continuing.
LLVM ERROR: Option 'default' already exists!

[harakas]

I found an env: AMD_COMGR_NAMESPACE=1 /opt/rocm/bin/migraphx-driver compile ... results in segfault:

Thread 24 "migraphx-driver" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe961ff700 (LWP 7957)]
0x00007ffea2b1eded in ?? ()
(gdb) bt
#0  0x00007ffea2b1eded in ?? ()
#1  0x0000000a31312d30 in ?? ()
#2  0x00007ffff148177b in llvm::sys::fs::openFileForRead(llvm::Twine const&, int&, llvm::sys::fs::OpenFlags, llvm::SmallVectorImpl<char>*) ()
   from /opt/rocm-therock.3/bin/../lib/migraphx/lib/libmigraphx_gpu.so.2013000
#3  0x0000000000000000 in ?? ()

[wsmoses]

@harakas do you have the build artifacts from your static build with the rock earlier, I'm curious if that means the XLA build I was working on gets running end to end

[harakas]

Sorry I can't share artifacts @wsmoses. But why not try to build it yourself? I'm building in a docker container without issues. It's rather trivial. Just need to find a beefy machine, and I recommend disabling fortran as well.

I have solved my linking problem on migraphx's side by:

diff --git a/src/targets/gpu/CMakeLists.txt b/src/targets/gpu/CMakeLists.txt
index 497ac403c..78b531ea2 100644
--- a/src/targets/gpu/CMakeLists.txt
+++ b/src/targets/gpu/CMakeLists.txt
@@ -254,6 +254,7 @@ if(MIGRAPHX_ENABLE_MLIR)
     # Make this private to avoid multiple inclusions of LLVM symbols.
     # TODO: Fix rocMLIR's library to hide LLVM internals.
     target_link_libraries(migraphx_gpu PRIVATE rocMLIR::rockCompiler)
+    target_link_options(migraphx_gpu PRIVATE "LINKER:--exclude-libs,librockCompiler.a")
 endif()
 
 if(MIGRAPHX_USE_HIPRTC)

Maybe this approach is an option for you as well.

[gkrivor]

Hi, to do not multiply issues I put it here.

Usage of MIGraphX + rocDecode produce same issue.

So, it isn't just a one point of misalignment, it looks like a global issue which prevents combining of different ROCm modules in one... "namespace". I didn't dig why, but on 6.4.4 version I was able to solve it just be building rocDecode and MIGraphX from sources, instead of usage libraries installed by apt install. But when I switched to 7+ version - building approach fails.

[harakas]

The LLVM symbol conflict issue in migraphx has been fixed: ROCm/AMDMIGraphX#4345

But there are more similar issues I've found and fixed in my own build (msgpack version conflict): ROCm/AMDMIGraphX#4366 (comment)

MIGraphX also has issues in how it embeds files. In my default migraphx build on therock resulted in obscure aborts. PR is here: ROCm/AMDMIGraphX#4319

[gkrivor]

@harakas , thanks for sharing!

The LLVM symbol conflict issue in migraphx has been fixed: ROCm/AMDMIGraphX#4345

Hmmm... Thought why I see issue using latest source... You made this option OFF by default, without it - fails on 7+, with -DMIGRAPHX_STRIP_SYMBOLS=ON fail caused by LLVM has gone in my case.