Appendix C: POCL

Fig. 17 OpenCL with discrete memory (Device: 2 processor+memory)

As Fig. 17, one possible HW platform has one Host and Device with 2 RISCV processors, each with dedicated memory.

• info::local_mem_type::global -> coherent, CPU cache (SRAM)

• info::local_mem_type::local -> dedicated local memory, GPU cache (SRAM)

OpenCL includes three components: Runtime API, Driver, and Compiler. POCL is an open-source implementation of the Runtime API.

OpenCL Implementation [1]. Wiki OpenCL Open Source Implementations [2].

OpenCL

Builtin-function and builtin-kernel reference here [6]. Books [7]. Papers [8] [9] [10] [11] [12] in lbt/papers/pocl.

Run OpenCL

MacOS

You can create the project ~/git/lbt/exlbt/opencl/opencl-saxpy/opencl-saxpy.xcodeproj by:

• Open Xcode

• Create a new Xcode project

• Select macOS → Command Line Tool

• Set Product Name: opencl-saxpy, Language: C

• Click Next → Create

• Choose folder ~/git/lbt/exlbt/opencl

• On the left, in the opencl-saxpy folder, delete the default main.c → Move to Trash

• Copy saxpy.c into ~/git/lbt/exlbt/opencl/opencl-saxpy/opencl-saxpy

• On the left, right-click opencl-saxpy folder → Add Files to “opencl-saxpy”

• Save the project

Build and run the saxpy OpenCL program:

• Open ~/git/lbt/exlbt/opencl/opencl-saxpy/opencl-saxpy.xcodeproj

• Choose Product → Build

• Choose Product → Run

Refer to the icons shown in Fig. 18 to see compile options and run results.

Fig. 18 Build and run opencl-saxpy.xcodeproj

Open sources

Table 8 Open sources for OpenCL

Project	Runtime	Driver	Compiler	Library
POCL	V	V
clang			V
libclc				V

Table 9 Open sources for OpenCL 2

project	Host Compiler	Device Compiler	Device Lib
POCL	X86	X86,ARM,AMD,TCE,PTX
clang	X86	NVIDIA-gpu, AMD-gpu
libclc			Spir, Spir64, NVIDIA-gpu, AMD-gpu

libclc is an open source, BSD/MIT dual licensed implementation of the library requirements of the OpenCL C programming language [3].

POCL

Pocl web [4] and documentation [5].

PoCL uses Clang as an OpenCL C frontend and LLVM for kernel compiler implementation, and as a portability layer. Thus, if your desired target has an LLVM backend, it should be able to get OpenCL support easily by using PoCL [4].

Note

OpenCL is a C-based language with the keyword __kernel and special attributes in data types. If Clang can compile input.cl to LLVM IR (input.ll), then the LLVM backend can compile input.ll and link it with libraries.

Build as the following bash:

exlbt/pocl/pocl-install.sh

#!/usr/bin/env bash

# add apt repo at first time
# sudo apt-add-repository 'deb https://apt.llvm.org/bionic/   llvm-toolchain-bionic-13 main'

# Set POCL_PARENT_DIR for the parent folder of pocl git hub checkout to.
export POCL_PARENT_DIR=$HOME/git

# After "sudo make install", pocl installed in /usr/local/share/pocl, 
# /usr/local/lib/libpocl.so, /usr/local/lib/pocl and /usr/local/bin/poclcc.

# Ubuntu 18.04 only can use LLVM_VERSION 13. 14 is too new for the dependent 
# packages of Ubuntu 18.04.
export LLVM_VERSION=13
export LLVM_PATH=/usr/lib/llvm-13/bin
#Not work for the following
#LLVM_PATH=$HOME/llvm/13/llvm-project/build/bin

# Ubuntu 22.04 only can use LLVM_VERSION 14. 13 has no /usr/lib/llvm-13/clang 
# after install_dependences().
#LLVM_VERSION=14
#LLVM_PATH=/usr/lib/llvm-14/bin

# Todo:
# Trace test_clCreateKernel.c and test_enqueue_kernel_from_binary.c for running an OpenCL example.

install_dependences() {
  echo "LLVM_VERSION: $LLVM_VERSION"
  sudo apt-get install -y build-essential ocl-icd-libopencl1 cmake git pkg-config \
  libclang-${LLVM_VERSION}-dev clang-${LLVM_VERSION} llvm-${LLVM_VERSION} make ninja-build \
  ocl-icd-libopencl1 ocl-icd-dev ocl-icd-opencl-dev libhwloc-dev zlib1g \
  zlib1g-dev clinfo dialog apt-utils libxml2-dev libclang-cpp${LLVM_VERSION}-dev \
  libclang-cpp${LLVM_VERSION} llvm-${LLVM_VERSION}-dev
}

get_pocl() {
  pushd $POCL_PARENT_DIR
  git clone https://github.com/pocl/pocl
  cd pocl
#  git checkout 4627171d40543091e399989c277faa52fcee0ff8
  popd
}

check() {
  if [ ! -d "$POCL_PARENT_DIR" ]; then
    echo "POCL_PARENT_DIR: $POCL_PARENT_DIR not exist"
    exit 1
  fi
}

build_pocl() {
  pushd $POCL_PARENT_DIR/pocl
  mkdir build
  cd build
# The default uses /usr/bin/cc in ubuntu 18.04
#  cmake -WITH_DLLVM_CONFIG=${LLVM_PATH}/llvm-config ..
# Have verified the following using clang compiler
  cmake -DWITH_LLVM_CONFIG=${LLVM_PATH}/llvm-config \
  -DENABLE_ICD=OFF \
  -DCMAKE_C_COMPILER=${LLVM_PATH}/clang \
  -DCMAKE_CXX_COMPILER=${LLVM_PATH}/clang++ ..
  make
  sudo make install
  popd
}

# Verify tests/runtime/test_clCreateKernel.c using clang rather than cc as follows,
# jonathanchen@hz-compiler1:~/git/pocl/build$ touch ../tests/runtime/test_clCreateKernel.c
# jonathanchen@hz-compiler1:~/git/pocl/build$ make VERBOSE=1 |grep test_clCreateKernel.c
# [ 95%] Building C object tests/runtime/CMakeFiles/test_clCreateKernel.dir/test_clCreateKernel.c.o
# cd /home/jonathanchen/git/pocl/build/tests/runtime && /usr/lib/llvm-13/bin/clang ...
# -c /home/jonathanchen/git/pocl/tests/runtime/test_clCreateKernel.c

# http://portablecl.org/docs/html/development.html
check_pocl() {
  pushd $POCL_PARENT_DIR/pocl/build
  make check_tier1
  popd
}

install_dependences;
get_pocl;
check;
build_pocl;
check_pocl;

Add apt repo on Ubuntu at first time,

$ sudo apt-add-repository 'deb https://apt.llvm.org/bionic/   llvm-toolchain-bionic-13 main'
$ grep "apt.llvm" /etc/apt/sources.list /etc/apt/sources.list.d/*
/etc/apt/sources.list:deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main
/etc/apt/sources.list:# deb-src http://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main
/etc/apt/sources.list:deb https://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main
/etc/apt/sources.list:# deb-src https://apt.llvm.org/bionic/ llvm-toolchain-bionic-13 main

Reference [13].

Structure

The runtime code is located in pocl/lib/CL. Test cases for the runtime can be found in pocl/tests/runtime.

GDB on POCL

After running bash pocl-install.sh, you can execute make check_tier1. However, running a single test such as ./tests/runtime/test_clCreateKernelsInProgram fails, as shown below:

~/git/pocl/build$ POCL_DEBUG=all ./tests/runtime/test_clCreateKernelsInProgram
...
[2022-07-15 08:43:44.140733258219579]POCL: in fn pocl_init_devices at line 529:
  |   WARNING |  cschen:Loading
  /home/cschen/git/pocl/build/lib/CL/pocl/libpocl-devices-basic.so failed:
  /home/cschen/git/pocl/build/lib/CL/pocl/libpocl-devices-basic.so: cannot
  open shared object file: No such file or directory
...
[2022-10-06 08:48:06.140725365544843]POCL: in fn pocl_init_devices at line 584:
  |     ERROR | CL_DEVICE_NOT_FOUND no devices found. POCL_DEVICES=(null)
CL_DEVICE_NOT_FOUND in poclu_get_any_device on line 22

Failing to run a single test executable is not acceptable for tracing the pocl code. A workaround fix is as follows:

~/git/pocl/build$ touch ../tests/runtime/test_clCreateKernelsInProgram.c
~/git/pocl/build$ make VERBOSE=1|grep test_clCreateKernelsInProgram
...
/usr/lib/llvm-13/bin/clang -g  -pie
CMakeFiles/test_clCreateKernelsInProgram.dir/test_clCreateKernelsInProgram.c.o
-o test_clCreateKernelsInProgram  -Wl,-rpath,/home/cschen/git/pocl/build/lib/CL
../../poclu/libpoclu.a ../../lib/CL/libOpenCL.so.2.9.0
-L/usr/lib/x86_64-linux-gnu -lhwloc /usr/lib/llvm-13/lib/libclang-cpp.so
/usr/lib/llvm-13/lib/libLLVM-13.so -lrt -lm -ldl -lm -ldl

Change the link path (rpath) from /home/cschen/git/pocl/build/lib/CL to /usr/local/lib for libpocl-devices-basic.so, then it passes as follows:

~/git/pocl/build$ cd /home/jonathanchen/git/pocl/build/tests/runtime
~/git/pocl/build/tests/runtime$ /usr/lib/llvm-13/bin/clang -g  -pie
CMakeFiles/test_clCreateKernelsInProgram.dir/test_clCreateKernelsInProgram.c.o
-o test_clCreateKernelsInProgram  -Wl,-rpath,/usr/local/lib
../../poclu/libpoclu.a ../../lib/CL/libOpenCL.so.2.9.0 -lhwloc
/usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/lib/libLLVM-13.so -lrt
-lm -ldl -lm -pthread -ldl
~/git/pocl/build/tests/runtime$ cd ../..
~/git/pocl/build$ ./tests/runtime/test_clCreateKernelsInProgram
Hello
World

Then I can use GDB as follows:

~/git/pocl/build/$ gdb --args ./tests/runtime/test_clCreateKernelsInProgram
(gdb) b test_clCreateKernelsInProgram.c:21
Breakpoint 1 at 0x23b4: file
/home/cschen/git/pocl/tests/runtime/test_clCreateKernelsInProgram.c, line 21.
(gdb) r
...
Breakpoint 1, main (argc=1, argv=0x7fffffffdfa8) at
/home/cschen/git/pocl/tests/runtime/test_clCreateKernelsInProgram.c:23
21        err = poclu_get_any_device(&ctx, &did, &queue);

Examples of Compiling and Running on POCL

As traced from Clang compilation options in the last section, I added a compile.sh script to run OpenCL programs on POCL as follows:

exlbt/pocl/ex/compile.sh

#!/usr/bin/env bash

FILE=test_clCreateKernelsInProgram
#FILE=saxpy

/usr/lib/llvm-13/bin/clang -DCL_HPP_TARGET_OPENCL_VERSION=300 -DCL_TARGET_OPENCL_VERSION=300 -DCL_USE_DEPRECATED_OPENCL_1_0_APIS -DCL_USE_DEPRECATED_OPENCL_1_1_APIS -DCL_USE_DEPRECATED_OPENCL_1_2_APIS -DCL_USE_DEPRECATED_OPENCL_2_0_APIS -DCL_USE_DEPRECATED_OPENCL_2_1_APIS -DCL_USE_DEPRECATED_OPENCL_2_2_APIS -I$HOME/git/pocl/build -I$HOME/git/pocl/include -I$HOME/git/pocl/include/hpp -I$HOME/git/pocl/poclu -I$HOME/git/pocl -g -fPIE -Werror=implicit-function-declaration -Wincompatible-pointer-types -Wno-ignored-attributes -fvisibility=hidden -pthread -MD -MT saxpy.c.o -MF $FILE.c.o.d -o $FILE.c.o -c $FILE.c

/usr/lib/llvm-13/bin/clang -g  -pie $FILE.c.o -o a.out  -Wl,-rpath,/usr/local/lib ~/git/pocl/build/poclu/libpoclu.a ~/git/pocl/build/lib/CL/libOpenCL.so.2.9.0 -lhwloc /usr/lib/llvm-13/lib/libclang-cpp.so /usr/lib/llvm-13/lib/libLLVM-13.so -lrt -lm -ldl -lm -pthread -ldl 

~/git/lbt/exlbt/pocl/ex$ POCL_DEBUG=err,warn
~/git/lbt/exlbt/pocl/ex$ bash compile.sh
~/git/lbt/exlbt/pocl/ex$ ./a.out
  ** Final POCL_DEBUG flags: 18000000000
  Hello World!
~/git/lbt/exlbt/pocl/ex$ POCL_DEBUG=
~/git/lbt/exlbt/pocl/ex$ ./a.out
  Hello World!

References [14].

RISCV OpenCL

In LLVM, both MIPS and Cpu0 can compile .ll files generated by Clang from OpenCL input. However, RISCV fails to do so on both LLVM 14.x and 15.x, as shown below:

CodeGenOpenCL % pwd
$HOME/git/lbt/exlbt/pocl
% ~/llvm/14.x/llvm-project/build/bin/clang -cc1 -cl-std=CL2.0 -emit-llvm
  -triple spir-unknown-unknown test.cl
% ~/llvm/debug/build/bin/llc -march=mips test.ll
% ~/llvm/test/build/bin/llc -march=cpu0 test.ll
% ~/llvm/14.x/llvm-project/build/bin/llc -mtriple=riscv64 test.ll
  LLVM ERROR: Unsupported calling convention
  ...
% ~/llvm/15.x/llvm-project/build/bin/clang -cc1 -cl-std=CL2.0 -emit-llvm -triple spir-unknown-unknown test.cl
% ~/llvm/15.x/llvm-project/build/bin/llc -mtriple=riscv64 test.ll
  LLVM ERROR: Unsupported calling convention
  ...

CodeGenOpenCL % pwd
$HOME/llvm/14.x/llvm-project/clang/test/CodeGenOpenCL
CodeGenOpenCL % ~/llvm/debug/build/bin/clang -cc1 -cl-std=CL2.0 -emit-llvm
               -triple spir-unknown-unknown overload.cl
CodeGenOpenCL % ~/llvm/debug/build/bin/llc -march=mips overload.ll
CodeGenOpenCL % ~/llvm/test/build/bin/llc -march=cpu0 overload.ll
CodeGenOpenCL % ~/llvm/14.x/llvm-project/build/bin/llc -mtriple=riscv64 overload.ll
                LLVM ERROR: Unsupported calling convention
                ...

SYCL

Webs [15]. An example [16].

DPC++

Book [17].

exlbt/pocl/dpc++.txt

[1]

https://www.iwocl.org/resources/opencl-implementations/

[2]

https://en.wikipedia.org/wiki/OpenCL#Open_source_implementations

[3]

https://libclc.llvm.org

[4] (1,2)

http://portablecl.org/

[5]

http://portablecl.org/docs/html/

[6]

http://jonathan2251.github.io/lbt/clang.html#builtin-functions

[7]

https://dahlan.unimal.ac.id/files/ebooks2/2015%203rd%20Heterogeneous%20Computing%20with%20OpenCL%202.0.pdf

[8]

https://mlsys.org/Conferences/doc/2018/84.pdf

[9]

https://www.researchgate.net/publication/350512301_Automatic_Graph_Partitioning_for_Very_Large-scale_Deep_Learning

[10]

https://easychair.org/publications/preprint/GjhX

[11]

https://www.researchgate.net/publication/221084751_A_comprehensive_performance_comparison_of_CUDA_and_OpenCL

[12]

https://arxiv.org/pdf/2002.03794.pdf

[13]

http://portablecl.org/docs/html/install.html#configure-build

[14]

http://portablecl.org/docs/html/debug.html

[15]

https://www.khronos.org/sycl

[16]

https://en.wikipedia.org/wiki/SYCL

[17]

https://itbook.store/books/9781484255735