Intel(R) SHMEM Library (ISHMEM) 1.3.0

Signed-off-by: sys_shmem <[email protected]>

Author: sys-shmem

.clang-format

@@ -22,3 +22,4 @@ IndentWidth:                                4
 KeepEmptyLinesAtTheStartOfBlocks:           false
 SortIncludes:                               false
 SpaceAfterCStyleCast:                       true
+NamespaceIndentation:                       All

.clang-format-ignore

@@ -0,0 +1,21 @@
+# Skip cmake files
+**/CMakeLists.txt
+**/cmake/*
+**/CMakeFiles/*
+
+# Skip non-source files/directories
+*.md
+LICENSE
+tags
+third-party-programs.txt
+.github/*
+.git/*
+docs/*
+maint/*
+scripts/*
+pkgconfig/*
+
+# Skip third-party files
+src/malloc.cpp
+src/uthash.h
+pmi-simple/*

.gitignore

@@ -1,3 +1,4 @@
 .*.sw*
 build
 tags
+ishmem/config.h

CMakeLists.txt

@@ -12,7 +12,7 @@ set(PROJECT_NAME "ishmem")
 set(PROJECT_FULL_NAME "Intel® SHMEM")
 
 set(ISHMEM_MAJOR_VERSION "1")
-set(ISHMEM_MINOR_VERSION "2")
+set(ISHMEM_MINOR_VERSION "3")
 set(ISHMEM_PATCH_VERSION "0")
 set(PROJECT_VERSION "${ISHMEM_MAJOR_VERSION}.${ISHMEM_MINOR_VERSION}.${ISHMEM_PATCH_VERSION}")
 

RELEASE_NOTES.md

@@ -1,6 +1,22 @@
 # Release Notes <!-- omit in toc -->
 This document contains a list of new features and known limitations of Intel® SHMEM releases.
 
+## Release 1.3.0
+
+### New Features and Enhancements
+- Support for C++ templated APIs for non-Debug build types.
+- Bug fixes improving functionality.
+
+### Known Limitations
+- Only [Sandia OpenSHMEM](https://github.com/Sandia-OpenSHMEM/SOS) and [Intel® MPI Library](https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html) are currently supported as the host back-end.
+- Not all APIs from OpenSHMEM standard are supported. Please refer to [Supported/Unsupported Features](https://oneapi-src.github.io/ishmem/supported_features.html) to get a complete view.
+- Intel® SHMEM requires a one-to-one mapping of PEs to SYCL devices. This implies that Intel® SHMEM executions must launch with a number of processes on each compute node that is no more than the number of available SYCL devices on each one of those nodes. By default, the Intel® SHMEM runtime considers each individual device tile to make up a single SYCL device and assigns a tile per PE.
+- All collective operations within a kernel must complete before invoking subsequent kernel-initiated collective operation.
+- To run Intel® SHMEM with SOS enabling the Slingshot provider in OFI, environment variable `FI_CXI_OPTIMIZED_MRS=0` must be used. It is also recommended to use `FI_CXI_DEFAULT_CQ_SIZE=131072`.
+- To run Intel® SHMEM with SOS enabling the verbs provider, environment variable `MLX5_SCATTER_TO_CQE=0` must be used.
+- To run Intel® SHMEM with Intel® MPI Library, environment variable `I_MPI_OFFLOAD=1` must be used. Additionally, `I_MPI_OFFLOAD_RDMA=1` may be necessary for GPU RDMA depending on the OFI provider. Please refer to the [reference guide](https://www.intel.com/content/www/us/en/docs/mpi-library/developer-reference-linux/2021-14/gpu-buffers-support.html) for further details.
+- Inter-node communication in Intel® SHMEM requires [dma-buf](https://www.kernel.org/doc/html/latest/driver-api/dma-buf.html) support in the Linux kernel. Inter-node functionality in Intel® SHMEM Release 1.3.0 is tested with SUSE Linux Enterprise Server 15 SP4.
+
 ## Release 1.2.0
 
 ### New Features and Enhancements

docs/source/compiling_and_running_programs.rst

@@ -44,3 +44,29 @@ Intel® SHMEM build configuration::
 
 See section :ref:`Library Constants<library_constants>` for more information
 about these variables.
+
+Selecting SPIR-V Compilation Targets
+------------------------------------
+
+On some systems, you may encounter an error in which the correct SPIR-V targets
+are not successfully selected when linking with Intel® SHMEM.  This may result in
+problems when using device-initiated communication including compilation
+warnings: ::
+
+    icpx: warning: linked binaries do not contain expected 'spir64-unknown-unknown' target; found targets: 'spir64_gen-unknown-unknown' [-Wsycl-target]
+
+as well as runtime errors: ::
+
+    terminate called after throwing an instance of 'sycl::_V1::compile_program_error'
+      what():  The program was built for 1 devices
+    Build program log for 'Intel(R) Data Center GPU Max 1550':
+    Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi>
+    Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi>
+    Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi>
+    Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi> -11 (PI_ERROR_BUILD_PROGRAM_FAILURE)
+
+This error can be resolved by indicating the desired target at compile time. To
+compile with the appropriate target for a Intel® Data Center GPU Max 1550 (PVC)
+GPU, add the following flags when linking: ::
+
+    -fsycl-targets=spir64_gen --start-no-unused-arguments -Xs "-device pvc" --end-no-unused-arguments --start-no-unused-arguments -Xsycl-target-backend "-q" --end-no-unused-arguments

docs/source/conf.py

@@ -16,7 +16,7 @@
 project = u'Intel® SHMEM'
 copyright = u'2024 Intel Corporation licensed under Creative Commons BY 4.0'
 author = u'Intel Corporation'
-release = u'1.2.0'
+release = u'1.3.0'
 version = release
 
 # -- General configuration ---------------------------------------------------

docs/source/library_setup_exit_query.rst

@@ -50,24 +50,27 @@ Callable from the **host**.
 Indicates which runtime is used to initialize Intel® SHMEM: either
 OpenSHMEM, MPI, or PMI.
 
-.. important:: As of Intel® SHMEM |version| only ISHMEM_RUNTIME_OPENSHMEM is supported.
+.. important:: As of Intel® SHMEM |version| only ISHMEM_RUNTIME_OPENSHMEM and ISHMEM_RUNTIME_MPI are supported.
 
 .. _ishmemx_attr_t:
 .. cpp:struct:: ishmemx_attr_t
 
   .. c:var:: ishmemx_runtime_type_t runtime
   .. c:var:: bool initialize_runtime = true
   .. c:var:: bool gpu = true
+  .. c:var:: void *mpi_comm
 
 **Description:**
 A struct declaration describing attributes for initialization.
 A valid **runtime** enumeration value must be set by the user and must
 correspond to a runtime that is enabled within the build of the ``ishmem``
 library.
-By default, the parallel runtime is assumed to be initialized by the user
-(**initialize_runtime** default is ``false``).
+By default, the parallel runtime is initialized by Intel® SHMEM
+(**initialize_runtime** default is ``true``).
 The **gpu** boolean indicates whether to use GPU memory for the symmetric
-heap (default is ``false``).
+heap (default is ``true``). **mpi_comm** is a pointer to the corresponding
+MPI communicator for representing ``ISHMEM_TEAM_WORLD`` when used with
+``ISHMEM_RUNTIME_MPI`` (default is ``MPI_COMM_WORLD``).
 
 .. _ishmemx_init_attr:
 .. cpp:function:: void ishmemx_init_attr(ishmemx_attr_t * attr)

docs/source/writing_programs.rst

@@ -4,9 +4,8 @@
 Writing Intel® SHMEM Programs
 ========================================
 
-Intel® SHMEM Programs require including the following header files::
+Intel® SHMEM Programs require including the following header file::
 
-#include <CL/sycl.hpp>
 #include <ishmem.h>
 
 Here is how to :ref:`initialize<library_setup_exit_query_routines>` the

examples/1_helloworld.cpp

@@ -2,7 +2,6 @@
  * SPDX-License-Identifier: BSD-3-Clause
  */
 
-#include <CL/sycl.hpp>
 #include <cstdlib>
 #include <iostream>
 #include <ishmem.h>