So you want to try HSA?

[DrMrLordX]

HSA isn't entirely vaporware. If you really want it badly enough, you can build the HSA software stack on your computer and run HSA-compliant software with it. Is it worth it to go through all that trouble? You be the judge. What I present here are (mostly) working instructions on how to run bonafide HSA software on your Kaveri processor, using the aparapi tree known as aparapi-lambda. These instructions will also enable you to run HSA binaries created with a combination of C/C++ and HSAIL instructions using Okra's C interface (okra.h) or possibly some other method.

First, read this thing:

https://code.google.com/p/aparapi/wiki/SettingUpLinuxHSAMachineForAparapi

It's out of date, and there are some problems with it, but it gives you an idea of what hurdles lie before you when setting up a machine to use aparapi-lambda. Essentially, you must install the HSA Drivers, the HSA Runtime, Okra, and then aparapi-lambda. There is more to it than clicking on some .exe installers since it involves Linux (and only Linux), and Linux is always an adventure. So, let's get started on a mostly up-to-date set of instructions on how to do this thing:

Pre-reqs

You must have the following hardware/software to get started:

A Kaveri processor. Other AMD APUs featuring GCN cores will not do. I'm looking at you, Kabini/Beema/Mullins! Carrizo will be supported in due time.

A motherboard that has working support for hardware IOMMU. Turn on IOMMU support whenever you want to run HSA software (turning it off won't really break anything, you just won't have a kfd which is the foundation for any HSA functionality).

Linux, preferably a Debian or rpm-based distro. My testbed was Lubuntu 14.10. Ubuntu 15.04 and variants may be better since they are based on 3.19 instead of 3.16. My instructions are based around a 64-bit distro so use that.

Linux kernel 3.19.

No Catalyst drivers (for now). Not counting betas, AMD has yet to release a Catalyst driver package for 3.19, and the patches out there are . . . well I wouldn't recommend it. It's up to you if you want to try adapting patches from the Arch Linux guys. The radeon drivers work just fine for HSA (so far as I can tell) so there's no harm in using those. AMD may fix this problem any day now, or they may just wait until 15.04 goes live next month for a Linux driver release.

Oracle JDK 8. You might be able to get some other JDK to work but I stuck with Oracle's version.

git, subversion, libelf-dev, build-essential, g++, and ant. Get those from your distro's repos or . . . wherever.

Some version of Gradle, installed in a directory named "gradle-1.12" somewhere on your machine (home directory is where I put mine). Unzip the -all or -binary distribution in that directory so that the bin directory is immediately accessible from gradle-1.12 and you should be fine. I used version 2.2.1 for what it's worth.

Setting up Okra

The next step is setting up Okra, which is a handy interface that allows one to access an HSA device via HSAIL. In order to do this, we must install the HSA Drivers, then the HSA Runtime, and finally Okra itself.

First, we have the HSA Drivers. Take a look at:

https://github.com/HSAFoundation/HSA-Drivers-Linux-AMD

You will observe that the latest driver package relies on a modified kernel based on 3.17.2, which we will not be using. We have to grab the two bits from this driver package that 3.19 does not give us: libhsakmt and the Kaveri HSA firmware.

Oded Gabbay was gracious enough to give me a link to a version of libhsakmt that functions with 3.19. Get it here:

git clone git://people.freedesktop.org/~gabbayo/libhsakmt

I ran that command from my home directory, fwiw. Once you've finished the clone operation, enter the libhsakmt directory and run type "make" in the console, and you're done. After that, we need the Kaveri firmware:

https://github.com/HSAFoundation/HS...er/kfd-1.0/ubuntu/radeon-firmware_391_all.deb for Debian-based distros or

https://github.com/HSAFoundation/HS...013-41.git0e5f6377.fc21.amdkfd.391.noarch.rpm for rpm-based distros

Install the package appropriate to your distro (if any) plus any dependences (I don't remember there being any but there might be) and then reboot. Nothing bad will happen from failure to reboot, but until you do, Okra will not work, and I'm pretty sure this package is what requires the reboot. But I digress.

Now we're ready to move on to the HSA Runtime. Take a look at this page:

https://github.com/HSAFoundation/HSA-Runtime-AMD

You can use git to clone the master build or you can just grab the .zip file (I grabbed the .zip). In either case, for simplicity, you should probably install this in your home directory as well (I did). Don't forget to install the appropriate package once you have built the directory tree:

hsa-runtime_1.0_amd64.deb for Debian-based distros and

hsa-runtime-1.0-0.fc21.x86_64.rpm for rpm-based distros

You can find both packages inside the HSA-Runtime-AMD-master directory (ubuntu or fedora subdirectories, as appropriate).

So far, so good, right? Great. Now let's go for Okra itself:

https://github.com/HSAFoundation/Okra-Interface-to-HSA-Device

Again, you can git clone the master or just download the zip, makes no difference (I grabbed the .zip). You should already have build-essential and libelf-dev installed as a pre-req so don't worry about those (if you don't, go install those now). Anyway, as before, create the directory structure inside your home directory (base directory should be Okra-Interface-to-HSA-Device-master). Then enter base Okra directory and run these two commands:

sudo cp okra/dist/bin/libokra_x86_64.so /opt/hsa/lib
sudo cp okra/dist/include/okra.h /opt/hsa/include

That copies libokra_x86_64.so and okra.h to the appropriate directories created when you installed libshakmt.

Then go to okra/samples and run build.sh to build the samples which are actually kind of helpful for validating your Okra install.

Assuming you rebooted sometime after installing the Kaveri firmware package, you should be able to use runSquares.sh in ~/Okra-Interface-to-HSA-Device-master/okra/samples/ without complication. If it works and you see numbers (and their squares) from 1 to 39, you're good to go for the following steps. If not, then heck if I know what's wrong but maybe if you post here we can sort it out. Or something!

Now, at this point, if you are comfortable with the idea of using a combination of C/C++ and HSAIL to produce HSA apps, you can stop now and forever hold your peace. That also assumes you don't want to run anyone's aparapi-lambda apps which (near as I can tell) require you to actually have aparapi-lambda installed to get them to run (as opposed to standard aparapi, which simply requires a copy of aparapi.jar to be distributed with the application). Here's the HSAIL code from Squares (the sample you just used to test your Okra install):

version 0:95: $full : $large;

kernel &run(
   kernarg_u64 %_out, 
   kernarg_u64 %_in
){
   ld_kernarg_u64 $d0, [%_out];
   ld_kernarg_u64 $d1, [%_in];
   
   @block0:
   workitemabsid_u32 $s2, 0;
   cvt_s64_s32 $d2, $s2;
   mad_u64 $d3, $d2, 4, $d1;
   ld_global_f32 $s3, [$d3];
   mad_u64 $d4, $d2, 4, $d1;
   ld_global_f32 $s4, [$d4];
   mul_f32 $s5, $s3, $s4;
   mad_u64 $d4, $d2, 4, $d0;
   st_global_f32 $s5, [$d4];
   ret;
   
};

If that floats your boat then hey, go for it. I uh, don't know HSAIL that well.

For those that want to continue on and use aparapi-lambda, we'll tackle the AMD APP SDK next!

AMD APP SDK

You don't HAVE to use AMD's SDK, but it's what I used, so I can not provide you with any other directions. Feel free to replace this step with something else if you know how to do so.

What we want is AMD APP SDK 2.9, not 3.0 beta. The reason for this is that 3.0 beta has an auto-installer that does "bad things" if it does not find Catalyst installed. Theoretically, you could install Catalyst and APP SDK 3.0 beta before installing kernel 3.19, then uninstall Catalyst and migrate to kernel 3.19 and hope that it worked, but I make no promises. You can always wait for Ubuntu 15.04 or just rip off the Catalyst 15.3 beta right now, but that's up to you.

AMD APP SDK 2.9 works, and it does not care whether you use Catalyst or the open-source radeon drivers. It might try to care, but it fails. Anyway, you can get version 2.9 here:

http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/

Scroll down and grab the 64-bit version. I have not looked at 2.9.1 to see if it has the crazy installer from 3.0 beta, but it probably does.

At this point, I highly recommend you follow the installation instructions from the original article from the aparapi google code wiki that I linked above. For clarity, I'll quote the relevant steps here:

Note that we have nested zipped jars in this archive.

$ cd ~
$ gunzip ~/Downloads/AMD-APP-SDK-v2.9-lnx64.tgz
$ tar xvf ~/Downloads/AMD-APP-SDK-v2.9-lnx64.tar
$ rm ~/default-install_lnx_64.pl ~/icd-registration.tgz ~/Install-AMD-APP.sh ~/ReadMe.txt
$ gunzip ~/AMD-APP-SDK-v2.9-RC-lnx64.tgz
$ tar xvf ~/AMD-APP-SDK-v2.9-RC-lnx64.tar
$ rm ~/AMD-APP-SDK-v2.9-RC-lnx64.tar
$ rm -rf AMD-APP-SDK-v2.9-RC-lnx64/samples

Note where AMD-APP-SDK-v2.9-RC-lnx64 is located, you need this in the following step.

I put AMD APP SDK in my home directory (just like everything else). Note that many of the instructions above involve getting rid of stuff that we don't need/want, which is something you can do easily with 2.9 and not-so-easily with 3.0 beta.

Now we're ready to move on to . . .

aparapi-lambda

Fair warning: this tree of aparapi has been stagnant since sometime in 2014, and it hasn't been migrated to aparapi's new GitHub page. Google is "sunsetting" Google Code which means that aparapi-lambda will become unavailable per the instructions I am about to provide below. Hopefully newer, less-buggy versions will become available at the GitHub page, saving us from losing easy access to aparapi-lambda.

That aside, let's grab aparapi-lambda via subversion:

svn checkout https://aparapi.googlecode.com/svn/branches/lambda aparapi-lambda

Again, I ran that from my home directory, so that's where it went.

Now, from here, you can try following the directions in the wiki article and hope for the best. I can add a few points that may help you here:

env.sh makes reference to GRADLE_HOME and prompts you to look for a directory called gradle-1.12. Fortunately, we should have that by now (check the pre-reqs) in our home directory, or wherever else you put it.

Using env.sh to make sure that all your environment variables match proper directories is a good idea, but it's a really good idea to set the variables by hand just in case (that helped me anyway). In particular, I needed to set JAVA_HOME, OKRA_HOME, APARAPI_HOME, OCL_HOME, OCL_LIB, OCL_INCLUDE, GRADLE_HOME, and ANT_HOME by hand. I also set up my .profile to set those variables at startup just in case I needed them again. Maybe that was going a bit too far? Maybe not. In any case, there are prompts inside the original env.sh file that should tell you which directories you need to specify when setting your variables. Since I put everything into my home directory that I could, my variables wound up like this (in no particular order):

GRADLE_HOME=/home/drmrlordx/gradle-1.12
ANT_HOME=/usr/share/ant
OKRA_HOME=/home/drmrlordx/Okra-Interface-to-HSA-Device-master/okra
JAVA_HOME=/usr/lib/jvm/java-8-oracle
OCL_HOME=/home/drmrlordx/AMD-APP-SDK-v2.9-RC-lnx64
OCL_LIB=/home/drmrlordx/AMD-APP-SDK-v2.9-RC-lnx64/lib/x86_64
OCL_INCLUDE=/home/drmrlordx/AMD-APP-SDK-v2.9-RC-lnx64/include
APARAPI_HOME=/home/drmrlordx/aparapi-lambda

And . . . that's it! After you run env.sh followed by ant, you should have aparapi-lambda built and functioning. Try running hsailmandel.sh in ~/aparapi-lambda/samples/mandel/ . If you can get that to run, you're good to go.

 

[DrMrLordX]

Now that you've got your lovely HSA/aparapi-lambda toolchain set up, what can you do with it outside of running the included samples?

Unless you are a Java programmer and are willing to deal with the bugs in aparapi-lambda . . . not a whole lot. I have not found any "native" HSAIL software (created via okra.h with hand-crafted HSAIL), nor have I found any Java software based on aparapi-lambda being distributed anywhere. Near as I can tell, it is not possible to run aparapi-lambda projects on anything but machines that have the entire toolchain described above, which is not a very practical model for software ditribution. At least with stuff like Squares, you can just compile that thing and run it on anything with a working kfd, HSA drivers, HSA runtime, and Okra. Somebody could probably write a script to pull down all the right stuff and install it auto-magically.

Someday soon-ish I'll release a version of Mathtester (HSAILMathtester) that has some HSA code in it, but it's kind of buggy, and I'm trying to figure out why hsailangleloop is causing segfaults and why hsailquadraticloop is bringing down the entire operating system (really). I may just have to throw in the towel on those since aparapi-lambda does some crazy stuff, for now.

But, even with that, that's like . . . one novelty program written by a forum zombie. I mean, seriously? What good is that? The worst part is that I stink at bash programming, so i can't figure out how to get my project to run unless I compile and run the code from the aparapi-lambda directory tree and use the provided env.sh. So sad. If I could figure out what env.sh is doing with 100% confidence, I could build my own shell script and make it run from wherever the enduser wanted (they'd still need aparapi-lambda installed somewhere, though).

If you're sneaky and know where to look, you can also use test versions of Project Sumatra, which involves building an entire custom JRE against Graal, but I haven't tried that yet, because the prospect of building the JRE like that scares me a little. It's probably not that hard but still.

 

[monstercameron]

I would totally try this but I just switched my system to an 860k...

 

[DrMrLordX]

D'oh! Oh well, no worries. Assuming you had any kind of dGPU in your APU system, some part of the toolchain might flip out (or it might not). It's a use-case I haven't examined yet. However! You could still try it on your FX-7500.

 

[DrMrLordX]

As mentioned above, here is an HSA-enabled application (HSAILMathtester):

https://www.dropbox.com/s/quz6s9g0rlu0bi8/HSAILMathtester.zip?dl=0

It's the same basic stuff as the 0327 build of Mathtester, except that it has HSAIL versions of all the CPU tests. The way things are set up right now, the HSAIL versions make no use of the x86 CPU cores for computation, relying instead on the GCN cores. Due to bugs in aparapi-lambda when using Device.hsa() from multiple threads, the HSA-enabled code only uses one thread and one task in the ExecutorService (as opposed to n threads and 48 tasks in the normal code, where n = the number of available CPU cores). It still performs ~103 billion iterations of the given operation (all of those iterations are assigned to a single task).

Here's how you use the stuff:

Download the zip and unzip it into your aparapi-lambda/samples directory (if you used your home dir, it should be ~/aparapi-lambda/samples). If you put it anywhere else, I make no promises that it will work. Option 5 from the main menu takes you to the HSA-enabled code. Note that autodetect for config.ini does NOT work for any of the HSAIL tests, and furthermore, you can not alter the config.ini settings for any HSAIL test using HSAILMathester's menu interface ('f' option from main menu). I recommend using the enclosed config.ini as a baseline for operating this version, especially when it comes to the HSA portion of the program. Run the program by typing:

./hsailmathtester.sh

If you wish to alter the settings in config.ini, you will have to do that by hand for the HSAIL tests. You can toggle "alternate" mode for a test by adding or removing the word Alternate or alternate (depends on the test, but you'll see the test names with capitol letters use Alternate and others use alternate) at the end of a test name.

The shell script is dependent on env.sh being where it is in the aparapi-lambda directory tree, so if you remove or rename env.sh then it won't work properly then, either.

You may periodically experience NullPointerExceptions or ArrayIndexOutOfBounds exceptions running the HSAIL tests, especially in batch mode. These problems are associated with flaws in aparapi-lambda and not reflective of any actual bug in HSAILMathester's code (well I think anyway). Sometimes the exceptions require the program to be ended and restarted, while sometimes they don't, and occasionally you'll get one of these errors without the current test stopping. You may wish to view results reported from erroneous tests with skepticism, and be advised that batch mode seems to make the errors more common. Occasionally you may experience a segfault (SIGSEGV 0xb). There's a chance that something in memory is corrupt (at least as far as aparapi-lambda's JNI code is concerned). You may need to reboot in order for HSAILMathtester to continue working properly. Hopefully I have identified all the causes of that behavior and eliminated them already.

To repeat, the HSA codepaths are 100% GPU-driven, aside from some CPU load to one core which is probably handling loop logic and such. Future versions of HSAILMathester will attempt to load-balance between the iGPU and CPU cores.

Like Mathtester, HSAILMathtester is under a BSD 2-clause license. I make no claim of copyright to any part of aparapi-lambda.