Appendix C: POCL

Fig. 17 OpenCL with discrete memory (Device: 2 processor+memory)

As Fig. 17, one possible HW platform is that one Host + Device with 2 RISCV processors and each RISCV has each dedicated memory.

• info::local_mem_type::global -> conherent, CPU cache (SRAM)

• info::local_mem_type::local -> dedicated local memory, GPU cache (SRAM)

OpenCL includes Runtime API, Driver and Compiler three components. POCL is an open source implementation for Runtime API.

OpenCL Implementation [1]. Wiki OpenCL Open source implementations [2].

OpenCL

Builtin-function and builtin-kernel reference here [6]. Books [7]. Papers [8] [9] [10] [11] [12] in lbt/papers/pocl.

Run OpenCL

Macos

~/git/lbt/exlbt/opencl/opencl-saxpy/opencl-saxpy.xcodeproj can be created by the following:

Xcode – Create a new Xcode project – macOS – Command Line Tool – Product Name: opencl-saxpy, Language: C – Next – Create – “Choose folder ~/git/lbt/exlbt/opencl” – on left opencl-saxpy folder: delete C main – Move to trash – “copy saxpy.c to ~/git/lbt/exlbt/opencl/opencl-saxpy/opencl-saxpy” – on left opencl-saxpy folder: right click mouse “Add files to “opencl-saxpy”” – save

Build and run saxpy opencl as follows,

Open ~/git/lbt/exlbt/opencl/opencl-saxpy/opencl-saxpy.xcodeproj – Product – Build – Product – Run, then choose the icons of the following Fig. 18 and get the compile options and run-result.

Fig. 18 Build and run opencl-saxpy.xcodeproj

Open sources

Table 8 Open sources for OpenCL

Project	Runtime	Driver	Compiler	Library
POCL	V	V
clang			V
libclc				V

Table 9 Open sources for OpenCL 2

project	Host Compiler	Device Compiler	Device Lib
POCL	X86	X86,ARM,AMD,TCE,PTX
clang	X86	NVIDIA-gpu, AMD-gpu
libclc			Spir, Spir64, NVIDIA-gpu, AMD-gpu

libclc is an open source, BSD/MIT dual licensed implementation of the library requirements of the OpenCL C programming language [3].

POCL

Pocl web [4] and documentation [5].

PoCL uses Clang as an OpenCL C frontend and LLVM for kernel compiler implementation, and as a portability layer. Thus, if your desired target has an LLVM backend, it should be able to get OpenCL support easily by using PoCL [4].

Note

OpenCL is C function with keyword __kernel and some attruibutes in data type. If clang is able to compile input.cl to input.ll, then llvm backend can compile input.ll and link to library.

Build as the following bash:

exlbt/pocl/pocl-install.sh

#!/usr/bin/env bash

# add apt repo at first time
# sudo apt-add-repository 'deb https://apt.llvm.org/bionic/   llvm-toolchain-bionic-13 main'

# Set POCL_PARENT_DIR for the parent folder of pocl git hub checkout to.
export POCL_PARENT_DIR=$HOME/git

# After "sudo make install", pocl installed in /usr/local/share/pocl, 
# /usr/local/lib/libpocl.so, /usr/local/lib/pocl and /usr/local/bin/poclcc.

# Ubuntu 18.04 only can use LLVM_VERSION 13. 14 is too new for the dependent 
# packages of Ubuntu 18.04.
export LLVM_VERSION=13
export LLVM_PATH=/usr/lib/llvm-13/bin
#Not work for the following
#LLVM_PATH=$HOME/llvm/13/llvm-project/build/bin

# Ubuntu 22.04 only can use LLVM_VERSION 14. 13 has no /usr/lib/llvm-13/clang 
# after install_dependences().
#LLVM_VERSION=14
#LLVM_PATH=/usr/lib/llvm-14/bin

# Todo:
# Trace test_clCreateKernel.c and test_enqueue_kernel_from_binary.c for running an OpenCL example.

install_dependences() {
  echo "LLVM_VERSION: $LLVM_VERSION"
  sudo apt-get install -y build-essential ocl-icd-libopencl1 cmake git pkg-config \
  libclang-${LLVM_VERSION}-dev clang-${LLVM_VERSION} llvm-${LLVM_VERSION} make ninja-build \
  ocl-icd-libopencl1 ocl-icd-dev ocl-icd-opencl-dev libhwloc-dev zlib1g \
  zlib1g-dev clinfo dialog apt-utils libxml2-dev libclang-cpp${LLVM_VERSION}-dev \
  libclang-cpp${LLVM_VERSION} llvm-${LLVM_VERSION}-dev
}

get_pocl() {
  pushd $POCL_PARENT_DIR
  git clone https://github.com/pocl/pocl
  cd pocl
#  git checkout 4627171d40543091e399989c277faa52fcee0ff8
  popd
}

check() {
  if [ ! -d "$POCL_PARENT_DIR" ]; then
    echo "POCL_PARENT_DIR: $POCL_PARENT_DIR not exist"
    exit 1
  fi
}

build_pocl() {
  pushd $POCL_PARENT_DIR/pocl
  mkdir build
  cd build
# The default uses /usr/bin/cc in ubuntu 18.04
#  cmake -WITH_DLLVM_CONFIG=${LLVM_PATH}/llvm-config ..
# Have verified the following using clang compiler
  cmake -DWITH_LLVM_CONFIG=${LLVM_PATH}/llvm-config \
  -DENABLE_ICD=OFF \
  -DCMAKE_C_COMPILER=${LLVM_PATH}/clang \
  -DCMAKE_CXX_COMPILER=${LLVM_PATH}/clang++ ..
  make
  sudo make install
  popd
}

# Verify tests/runtime/test_clCreateKernel.c using clang rather than cc as follows,
# jonathanchen@hz-compiler1:~/git/pocl/build$ touch ../tests/runtime/test_clCreateKernel.c
# jonathanchen@hz-compiler1:~/git/pocl/build$ make VERBOSE=1 |grep test_clCreateKernel.c
# [ 95%] Building C object tests/runtime/CMakeFiles/test_clCreateKernel.dir/test_clCreateKernel.c.o
# cd /home/jonathanchen/git/pocl/build/tests/runtime && /usr/lib/llvm-13/bin/clang ...
# -c /home/jonathanchen/git/pocl/tests/runtime/test_clCreateKernel.c

# http://portablecl.org/docs/html/development.html
check_pocl() {
  pushd $POCL_PARENT_DIR/pocl/build
  make check_tier1
  popd
}

install_dependences;
get_pocl;
check;
build_pocl;
check_pocl;

Add apt repo on Ubuntu at first time,

$ sudo apt-add-repository 'deb https://apt.llvm.org/bionic/   llvm-toolchain-bionic-13 main'
$ grep "apt.llvm" /etc/apt/sources.list /etc/apt/sources.list.d/*
/etc/apt/sources.list:deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main
/etc/apt/sources.list:# deb-src http://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main
/etc/apt/sources.list:deb https://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main
/etc/apt/sources.list:# deb-src https://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main

Reference [13].

Structure

Code of Runtime in pocl/lib/CL. Test cases of Runtime in pocl/tests/runtime.

GDB on POCL

After bash pocl-install.sh, can run make check_tier1 but fail on run single test such as ./tests/runtime/test_clCreateKernelsInProgram as follows,

~/git/pocl/build$ POCL_DEBUG=all ./tests/runtime/test_clCreateKernelsInProgram
...
[2022-07-15 08:43:44.140733258219579]POCL: in fn pocl_init_devices at line 529:
  |   WARNING |  cschen:Loading /home/cschen/git/pocl/build/lib/CL/pocl/libpocl-devices-basic.so failed: /home/cschen/git/pocl/build/lib/CL/pocl/libpocl-devices-basic.so: cannot open shared object file: No such file or directory
...
[2022-10-06 08:48:06.140725365544843]POCL: in fn pocl_init_devices at line 584:
  |     ERROR | CL_DEVICE_NOT_FOUND no devices found. POCL_DEVICES=(null)
CL_DEVICE_NOT_FOUND in poclu_get_any_device on line 22

Fail to run single test of executable file is not accpetable for tracing code of pocl. Workaround fix as follows,

~/git/pocl/build$ touch ../tests/runtime/test_clCreateKernelsInProgram.c
~/git/pocl/build$ make VERBOSE=1|grep test_clCreateKernelsInProgram
...
/usr/lib/llvm-13/bin/clang -g  -pie CMakeFiles/test_clCreateKernelsInProgram.dir/test_clCreateKernelsInProgram.c.o -o test_clCreateKernelsInProgram  -Wl,-rpath,/home/cschen/git/pocl/build/lib/CL ../../poclu/libpoclu.a ../../lib/CL/libOpenCL.so.2.9.0 -L/usr/lib/x86_64-linux-gnu -lhwloc /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/lib/libLLVM-13.so -lrt -lm -ldl -lm -ldl

Change link path rpath from /home/cschen/git/pocl/build/lib/CL to /usr/local/lib for libpocl-devices-basic.so then PASS as follows,

~/git/pocl/build$ cd /home/jonathanchen/git/pocl/build/tests/runtime
~/git/pocl/build/tests/runtime$ /usr/lib/llvm-13/bin/clang -g  -pie CMakeFiles/test_clCreateKernelsInProgram.dir/test_clCreateKernelsInProgram.c.o -o test_clCreateKernelsInProgram  -Wl,-rpath,/usr/local/lib ../../poclu/libpoclu.a ../../lib/CL/libOpenCL.so.2.9.0 -lhwloc /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/lib/libLLVM-13.so -lrt -lm -ldl -lm -pthread -ldl
~/git/pocl/build/tests/runtime$ cd ../..
~/git/pocl/build$ ./tests/runtime/test_clCreateKernelsInProgram
Hello
World

Then I can use gdb as follows,

~/git/pocl/build/$ gdb --args ./tests/runtime/test_clCreateKernelsInProgram
(gdb) b test_clCreateKernelsInProgram.c:21
Breakpoint 1 at 0x23b4: file /home/cschen/git/pocl/tests/runtime/test_clCreateKernelsInProgram.c, line 21.
(gdb) r
...
Breakpoint 1, main (argc=1, argv=0x7fffffffdfa8) at /home/cschen/git/pocl/tests/runtime/test_clCreateKernelsInProgram.c:23
21        err = poclu_get_any_device(&ctx, &did, &queue);

Examples of Compiling and running on POCL

As tracing clang compilation options in last section, I add compile.sh for running OpenCl program on pocl as follows,

exlbt/pocl/ex/compile.sh

#!/usr/bin/env bash

FILE=test_clCreateKernelsInProgram
#FILE=saxpy

/usr/lib/llvm-13/bin/clang -DCL_HPP_TARGET_OPENCL_VERSION=300 -DCL_TARGET_OPENCL_VERSION=300 -DCL_USE_DEPRECATED_OPENCL_1_0_APIS -DCL_USE_DEPRECATED_OPENCL_1_1_APIS -DCL_USE_DEPRECATED_OPENCL_1_2_APIS -DCL_USE_DEPRECATED_OPENCL_2_0_APIS -DCL_USE_DEPRECATED_OPENCL_2_1_APIS -DCL_USE_DEPRECATED_OPENCL_2_2_APIS -I$HOME/git/pocl/build -I$HOME/git/pocl/include -I$HOME/git/pocl/include/hpp -I$HOME/git/pocl/poclu -I$HOME/git/pocl -g -fPIE -Werror=implicit-function-declaration -Wincompatible-pointer-types -Wno-ignored-attributes -fvisibility=hidden -pthread -MD -MT saxpy.c.o -MF $FILE.c.o.d -o $FILE.c.o -c $FILE.c

/usr/lib/llvm-13/bin/clang -g  -pie $FILE.c.o -o a.out  -Wl,-rpath,/usr/local/lib ~/git/pocl/build/poclu/libpoclu.a ~/git/pocl/build/lib/CL/libOpenCL.so.2.9.0 -lhwloc /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/lib/libLLVM-13.so -lrt -lm -ldl -lm -pthread -ldl 

~/git/lbt/exlbt/pocl/ex$ POCL_DEBUG=err,warn
~/git/lbt/exlbt/pocl/ex$ bash compile.sh
~/git/lbt/exlbt/pocl/ex$ ./a.out
** Final POCL_DEBUG flags: 18000000000
Hello World!
~/git/lbt/exlbt/pocl/ex$ POCL_DEBUG=
~/git/lbt/exlbt/pocl/ex$ ./a.out
Hello World!

References [14].

RISCV OpenCL

In llvm, Mips and Cpu0 can compile .ll file from clang’s output for input OpenCL while RISCV fail on both 14.x and 15.x as follows,

CodeGenOpenCL % pwd
$HOME/git/lbt/exlbt/pocl
% ~/llvm/14.x/llvm-project/build/bin/clang -cc1 -cl-std=CL2.0 -emit-llvm -triple spir-unknown-unknown test.cl
% ~/llvm/debug/build/bin/llc -march=mips test.ll
% ~/llvm/test/build/bin/llc -march=cpu0 test.ll
% ~/llvm/14.x/llvm-project/build/bin/llc -mtriple=riscv64 test.ll
LLVM ERROR: Unsupported calling convention
...
% ~/llvm/15.x/llvm-project/build/bin/clang -cc1 -cl-std=CL2.0 -emit-llvm -triple spir-unknown-unknown test.cl
% ~/llvm/15.x/llvm-project/build/bin/llc -mtriple=riscv64 test.ll
LLVM ERROR: Unsupported calling convention
...

CodeGenOpenCL % pwd
$HOME/llvm/14.x/llvm-project/clang/test/CodeGenOpenCL
CodeGenOpenCL % ~/llvm/debug/build/bin/clang -cc1 -cl-std=CL2.0 -emit-llvm -triple spir-unknown-unknown overload.cl
CodeGenOpenCL % ~/llvm/debug/build/bin/llc -march=mips overload.ll
CodeGenOpenCL % ~/llvm/test/build/bin/llc -march=cpu0 overload.ll
CodeGenOpenCL % ~/llvm/14.x/llvm-project/build/bin/llc -mtriple=riscv64 overload.ll
LLVM ERROR: Unsupported calling convention
...

SYCL

Webs [15]. An example [16].

DPC++

Book [17].

exlbt/pocl/dpc++.txt

[1]

https://www.iwocl.org/resources/opencl-implementations/

[2]

https://en.wikipedia.org/wiki/OpenCL#Open_source_implementations

[3]

https://libclc.llvm.org

[4] (1,2)

http://portablecl.org/

[5]

http://portablecl.org/docs/html/

[6]

http://jonathan2251.github.io/lbt/clang.html#builtin-functions

[7]

https://dahlan.unimal.ac.id/files/ebooks2/2015%203rd%20Heterogeneous%20Computing%20with%20OpenCL%202.0.pdf

[8]

https://mlsys.org/Conferences/doc/2018/84.pdf

[9]

https://www.researchgate.net/publication/350512301_Automatic_Graph_Partitioning_for_Very_Large-scale_Deep_Learning

[10]

https://easychair.org/publications/preprint/GjhX

[11]

https://www.researchgate.net/publication/221084751_A_comprehensive_performance_comparison_of_CUDA_and_OpenCL

[12]

https://arxiv.org/pdf/2002.03794.pdf

[13]

http://portablecl.org/docs/html/install.html#configure-build

[14]

http://portablecl.org/docs/html/debug.html

[15]

https://www.khronos.org/sycl

[16]

https://en.wikipedia.org/wiki/SYCL

[17]

https://itbook.store/books/9781484255735