Allow querying free/total memory for SYCL by masterleinad · Pull Request #1225 · kokkos/kokkos-kernels

masterleinad · 2021-12-14T18:01:22Z

This is an intermediate solution for #1062 until the functionality is properly supported in the compiler.
Currently, there are some issues with using Sysman.

kokkos-devops-admin · 2021-12-14T18:09:55Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

masterleinad · 2021-12-16T21:17:45Z

This works reasonably well for me as a workaround. Unfortunately, we currently need to build with shared libraries for which I have introduced another CMake variable and we also have to link with ze_loader which this pull request just assumes to be accessible to the linker.

@brian-kelley Would you mind checking if that works for you running, e.g. wiki_spgemm on an Intel GPU?

kokkos-devops-admin · 2021-12-16T21:18:54Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

brian-kelley · 2021-12-17T00:50:13Z

@masterleinad Yes, that worked, thanks! The only caveat was I had to do export ZES_ENABLE_SYSMAN=1 before running.

srajama1

@masterleinad : Is there some reference for this memory module business?

masterleinad · 2021-12-17T21:50:07Z

@masterleinad : Is there some reference for this memory module business?

See https://spec.oneapi.io/level-zero/latest/sysman/api.html#zesdeviceenummemorymodules and https://spec.oneapi.io/level-zero/latest/sysman/api.html#zesmemorygetproperties. It's hard to find precise information and I would consider the functionality provided here fairly brittle but it seems to work well enough for the tests to pass until https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/LevelZeroBackend/LevelZeroBackend.md#51-device-information-descriptors is actually implemented in the compiler.

masterleinad · 2022-01-07T14:44:20Z

It would be great if we can think about this being considered for the next release.

kokkos-devops-admin · 2022-01-07T17:35:06Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

Just needs kokkos#1225 to allow querying free/total memory. This will make spgemm, spgemm-jacobi and block GS tests pass. replaceSumIntoLonger still has a runtime error (too many registers used by arrays?), so that is disabled for now.

lucbv · 2022-01-18T21:52:50Z

@masterleinad would you mind rebasing on top of develop?
After a quick review, I am happy to give the pre-approved label to have the CI pick it up.
@srajama1 is also looking at it today : )

masterleinad · 2022-01-18T22:02:23Z

Rebased.

masterleinad · 2022-01-18T22:04:14Z

+    IF (NOT BUILD_SHARED_LIBS)
+      MESSAGE(SEND_ERROR
+        "At the moment, we require KokkosKernels (and Kokkos) to be built as "
+        "shared libraries to allow querying free and total device memory!"
+      )
+    ENDIF()


Note that this is a new restriction that may surprise users already using SYCL.

lucbv · 2022-01-18T22:44:35Z

 INCLUDE(cmake/fake_tribits.cmake)
 INCLUDE(cmake/kokkoskernels_tribits.cmake)

+OPTION(BUILD_SHARED_LIBS "Build shared libraries" OFF)


This will change the default build behavior of Kokkos Kernels which does a shared library build by default.
Is this change required for your fix?

Oh, really? In fact, this patch requires building with shared libraries, see https://github.com/masterleinad/kokkos-kernels/blob/5a8e0146189c898551662b0ac9b1e2c580bfaa14/src/CMakeLists.txt#L470. I can check tomorrow if the on;y problem was that Kokkos didn't get built as a shared library.

Actually after double checking, we are building static libraries by default so this line should not change anything.

lucbv

looks to me

kokkos-devops-admin · 2022-01-18T23:21:59Z

Status Flag 'Pre-Test Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED by label AT: PRE-TEST INSPECTED! Autotester is Removing Label; This inspection will remain valid until a new commit to source branch is performed.

kokkos-devops-admin · 2022-01-18T23:26:31Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Build Num: 125
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Build Num: 123
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Build Num: 781
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Build Num: 428
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Build Num: 772
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Build Num: 760
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Build Num: 165
Status: STARTED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Using Repos:

Repo: KOKKOSKERNELS (masterleinad/kokkos-kernels)

Pull Request Author: masterleinad

kokkos-devops-admin · 2022-01-19T00:42:45Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Build Num: 125
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Build Num: 123
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Build Num: 781
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Build Num: 428
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Build Num: 772
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Build Num: 760
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Build Num: 165
Status: PASSED

Jenkins Parameters

Parameter Name	Value
KOKKOSKERNELS_SOURCE_BRANCH	query_memory_sycl
KOKKOSKERNELS_SOURCE_REPO	https://github.com/masterleinad/kokkos-kernels
KOKKOSKERNELS_SOURCE_SHA	`5a8e014`
KOKKOSKERNELS_TARGET_BRANCH	develop
KOKKOSKERNELS_TARGET_REPO	https://github.com/kokkos/kokkos-kernels
KOKKOSKERNELS_TARGET_SHA	`86f783a`
PR_LABELS
PULLREQUESTNUM	1225
TEST_REPO_ALIAS	KOKKOSKERNELS

kokkos-devops-admin · 2022-01-19T00:43:03Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ lucbv ]!

kokkos-devops-admin · 2022-01-19T00:43:10Z

Status Flag 'Pull Request AutoTester' - Pull Request MUST BE MERGED MANUALLY BY Project Team - This Repo does not support Automerge

sivascience

Thanks @masterleinad ! Looks good to me.

Just needs kokkos#1225 to allow querying free/total memory. This will make spgemm, spgemm-jacobi and block GS tests pass. replaceSumIntoLonger still has a runtime error (too many registers used by arrays?), so that is disabled for now.

* Get sparse tests enabled and working for SYCL Just needs #1225 to allow querying free/total memory. This will make spgemm, spgemm-jacobi and block GS tests pass. replaceSumIntoLonger still has a runtime error (too many registers used by arrays?), so that is disabled for now. * Added --shared option to cm_generate_makefile to build Kokkos and KokkosKernels as shared libraries. Static libraries are still the default. This option is useful because SYCL builds on Intel now require a shared library build.

masterleinad marked this pull request as ready for review December 16, 2021 21:17

masterleinad force-pushed the query_memory_sycl branch from 985dcb0 to 5fd3790 Compare December 16, 2021 21:34

masterleinad mentioned this pull request Dec 17, 2021

[SYCL][DOC] Initial Draft of Extension for querying free device memory on Level Zero intel/llvm#3468

Merged

srajama1 reviewed Dec 17, 2021

View reviewed changes

masterleinad force-pushed the query_memory_sycl branch from 5fd3790 to 0a692fb Compare January 7, 2022 14:42

brian-kelley mentioned this pull request Jan 17, 2022

Get sparse tests enabled and working for SYCL #1269

Merged

masterleinad and others added 3 commits January 18, 2022 17:00

Allow querying free/total memory for SYCL

1563327

Add option to build shared libraries

49a239f

Link with ze_loader when using SYCL+Intel GPUs(AOT)

5a8e014

masterleinad force-pushed the query_memory_sycl branch from 0a692fb to 5a8e014 Compare January 18, 2022 22:01

masterleinad commented Jan 18, 2022

View reviewed changes

lucbv requested changes Jan 18, 2022

View reviewed changes

lucbv approved these changes Jan 18, 2022

View reviewed changes

lucbv added the AT: PRE-TEST INSPECTED label Jan 18, 2022

kokkos-devops-admin removed the AT: PRE-TEST INSPECTED label Jan 18, 2022

sivascience approved these changes Jan 19, 2022

View reviewed changes

lucbv merged commit 701fb05 into kokkos:develop Jan 19, 2022

brian-kelley mentioned this pull request May 11, 2026

Error: memory space SYCLDeviceUSM does not support querying free/total memory. #1357

Closed

Conversation

masterleinad commented Dec 14, 2021

Uh oh!

kokkos-devops-admin commented Dec 14, 2021

Uh oh!

masterleinad commented Dec 16, 2021

Uh oh!

kokkos-devops-admin commented Dec 16, 2021

Uh oh!

brian-kelley commented Dec 17, 2021

Uh oh!

srajama1 left a comment

Choose a reason for hiding this comment

Uh oh!

masterleinad commented Dec 17, 2021

Uh oh!

masterleinad commented Jan 7, 2022

Uh oh!

kokkos-devops-admin commented Jan 7, 2022

Uh oh!

lucbv commented Jan 18, 2022

Uh oh!

masterleinad commented Jan 18, 2022

Uh oh!

masterleinad Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucbv Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

masterleinad Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucbv Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

lucbv left a comment

Choose a reason for hiding this comment

Uh oh!

kokkos-devops-admin commented Jan 18, 2022

Uh oh!

kokkos-devops-admin commented Jan 18, 2022

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_INTEL18

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_CLANG1001

Jenkins Parameters

Using Repos:

Uh oh!

kokkos-devops-admin commented Jan 19, 2022

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA9_GCC720_Light_Tpls_GCC720_GCC740

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_CUDA10_Tpls_CUDA10_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_GCC720_Light_LayoutRight

Jenkins Parameters

Build Information

Test Name: KokkosKernels_PullRequest_Tpls_GCC720