Info 12th BOINC Pentathlon 2021

[Fardringle]

480 PPS WU stuck in pending since 0Z, this normal?

Yup. Out of about 1000 PPS tasks that I returned starting just a few minutes after the start of the event, more than 600 are still pending validation. Just have to wait for the second person to complete and return their results.

 

[StefanR5R]

I added a few bits of project information to post #1, yesterday for PrimeGrid/ City Run and just now for Einstein@Home/ Cross Country.

 

[pipin]

Your PrimeGrid output is reaching sick levels.

 

[StefanR5R]

Your PrimeGrid output is reaching sick levels.

Yeah, my team mates are reckless, speeding in the middle of the city.

I am managing the cooling by means of open windows, as usual. [...] And until today, strong wind helped to carry waste heat out of the apartment too.

Now that he wind has let off, aspiring bee queens are checking out my apartment methodically, looking for spaces to lodge. I need to put up signs that I don't sublet.

 

[StefanR5R]

Bunkers, what bunkers?

BTW one of my secret plans for this Pentathlon was to spy on you in your forum. But I gave up early — too much traffic! :-O

 

[lane42]

Overall Standings
TAAT are in #1, and clearly at that. This is the surprising current standing. Even though Javelin Throw skews the result, the lead is nevertheless well earned,

 

[Skillz]

Overall Standings
TAAT are in #1, and clearly at that. This is the surprising current standing. Even though Javelin Throw skews the result, the lead is nevertheless well earned,

That will only stand if we can secure 2nd in the throw. Then, things will get more complicated once the next two disciplines finish.

edit
Well technically P3D will take the tie-breaker since they have more gold medals.

 

[lane42]

For those in the know, what projects should we be attacking now.
iam sure some are lost, Like me

 

[Endgame124]

For those in the know, what projects should we be attacking now.
iam sure some are lost, Like me

The easy / safe answer is sidock for CPU and Einstein for GPU.

 

[Skillz]

Correct.

Put CPUs on SiDock and Einstein on GPUs.
If you participated in any of the Javelin (WCG - MIP) throws then please do the same amount of work + 10% as you did on the others.

 

[Endgame124]

@Endgame124, post the startup part of the client log and the entire coproc_info.xml, here or somewhere befitting.

Startup portion:
Mon 10 May 2021 09:08:32 AM EDT | | Starting BOINC client version 7.16.6 for x86_64-pc-linux-gnu
Mon 10 May 2021 09:08:32 AM EDT | | log flags: file_xfer, sched_ops, task, coproc_debug
Mon 10 May 2021 09:08:32 AM EDT | | Libraries: libcurl/7.65.3 OpenSSL/1.1.1g-fips zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.9.3/openssl/zlib nghttp2/1.40.0
Mon 10 May 2021 09:08:32 AM EDT | | Data directory: /var/lib/boinc
Mon 10 May 2021 09:08:32 AM EDT | | [coproc] launching child process at /usr/bin/boinc
Mon 10 May 2021 09:08:32 AM EDT | | [coproc] with data directory /var/lib/boinc
Mon 10 May 2021 09:08:32 AM EDT | | OpenCL CPU: pthread-AMD Ryzen 7 2700X Eight-Core Processor (OpenCL driver vendor: The pocl project, driver version 1.3-pre, device version OpenCL 1.2 pocl HSTR: pthread-x86_64-unknown-linux-gnu-znver1)
Mon 10 May 2021 09:08:32 AM EDT | | [coproc] NVIDIA drivers present but no GPUs found
Mon 10 May 2021 09:08:32 AM EDT | | [coproc] ATI: libaticalrt.so: cannot open shared object file: No such file or directory
Mon 10 May 2021 09:08:32 AM EDT | | No usable GPUs found
Mon 10 May 2021 09:08:32 AM EDT | | libc: GNU libc version 2.29
Mon 10 May 2021 09:08:32 AM EDT | | Host name: Ares
Mon 10 May 2021 09:08:32 AM EDT | | Processor: 16 AuthenticAMD AMD Ryzen 7 2700X Eight-Core Processor [Family 23 Model 8 Stepping 2]
Mon 10 May 2021 09:08:32 AM EDT | | Processor features: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
Mon 10 May 2021 09:08:32 AM EDT | | OS: Linux Fedora: Fedora 30 (Workstation Edition) [5.6.13-100.fc30.x86_64|libc 2.29 (GNU libc)]
Mon 10 May 2021 09:08:32 AM EDT | | Memory: 31.34 GB physical, 22.84 GB virtual
Mon 10 May 2021 09:08:32 AM EDT | | Disk: 68.40 GB total, 32.56 GB free
Mon 10 May 2021 09:08:32 AM EDT | | Local time is UTC -4 hours


cat coproc_info.xml
<coprocs>
<opencl_cpu_prop>
<platform_vendor>The pocl project</platform_vendor>
<opencl_cpu_info>
<name>pthread-AMD Ryzen 7 2700X Eight-Core Processor</name>
<vendor>AuthenticAMD</vendor>
<vendor_id>1818455920</vendor_id>
<available>1</available>
<half_fp_config>0</half_fp_config>
<single_fp_config>191</single_fp_config>
<double_fp_config>191</double_fp_config>
<endian_little>1</endian_little>
<execution_capabilities>3</execution_capabilities>
<extensions>cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_3d_image_writes cl_khr_fp64 cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64</extensions>
<global_mem_size>31506845696</global_mem_size>
<local_mem_size>4194304</local_mem_size>
<max_clock_frequency>3700</max_clock_frequency>
<max_compute_units>16</max_compute_units>
<nv_compute_capability_major>0</nv_compute_capability_major>
<nv_compute_capability_minor>0</nv_compute_capability_minor>
<amd_simd_per_compute_unit>0</amd_simd_per_compute_unit>
<amd_simd_width>0</amd_simd_width>
<amd_simd_instruction_width>0</amd_simd_instruction_width>
<opencl_platform_version>OpenCL 1.2 pocl 1.3-pre RelWithDebInfo, LLVM 8.0.0, SLEEF, DIST</opencl_platform_version>
<opencl_device_version>OpenCL 1.2 pocl HSTR: pthread-x86_64-unknown-linux-gnu-znver1</opencl_device_version>
<opencl_driver_version>1.3-pre</opencl_driver_version>
</opencl_cpu_info>
</opencl_cpu_prop>
<warning>NVIDIA drivers present but no GPUs found</warning>
<warning>ATI: libaticalrt.so: cannot open shared object file: No such file or directory</warning>
</coprocs>


lspci | grep VGA
09:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)

 

[Icecold]

Which Nvidia driver version is it using and is it the closed source drivers?

 

[Endgame124]

Which Nvidia driver version is it using and is it the closed source drivers?

Its 440 from rpmfusion:


sudo dnf list installed | grep -i nvidia
akmod-nvidia.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
kmod-nvidia-5.5.10-100.fc30.x86_64.x86_64 3:440.82-1.fc30 @@commandline
kmod-nvidia-5.6.13-100.fc30.x86_64.x86_64 3:440.82-1.fc30 @@commandline
nvidia-modprobe.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
nvidia-persistenced.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
nvidia-query-resource-opengl-lib.i686 1.0.0-5.fc30 @fedora
nvidia-settings.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
nvidia-texture-tools.i686 2.0.8-20.fc30 @fedora
nvidia-texture-tools-devel.i686 2.0.8-20.fc30 @fedora
nvidia-xconfig.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.i686 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-cuda-libs.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-kmodsrc.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-libs.i686 3:440.82-1.fc30 @rpmfusion-nonfree-updates
xorg-x11-drv-nvidia-libs.x86_64 3:440.82-1.fc30 @rpmfusion-nonfree-updates

 

[Endgame124]

In an also extremely frustrating turn of events on getting multiple Einstein WUs to run in parallel

I set the web preferences to .5 GPU, and my main machine still doesn't recognize it - it won't run multiple WUs regardless of app_config.xml or web prefs. On this host, I do have the following under computing preference:
Using web-based preferences from:
www worldcommunitygrid com

Has WCG messed up boinc on this system?

Additionally, I turned on some old hardware to run Einstein with a RX 560 and an old AMD APU. It respects the web preferences and immediately started running 2 tasks in parallel. I switched the profile in the Einstein website, and its STILL running WUs in parallel.

 

[biodoc]

F@H runs Cuda WUs - runs about 2.4M ppd.

Boinc sees the drivers - the startup log shows something roughly like “NVIDIA GPU drivers found, but no usable GPUs”.

It's possible you need an opencl loader and development files.

For example on Mint/Ubuntu, we need to install 2 packages to get opencl to work on nvidia GPUs:

sudo apt install ocl-icd-libopencl1
sudo apt install ocl-icd-opencl-dev

I'm not sure which RPM packages to install on your version of Fedora but I did an RPM search for opencl here.

 

[Endgame124]

It's possible you need an opencl loader and development files.

For example on Mint/Ubuntu, we need to install 2 packages to get opencl to work on nvidia GPUs:

sudo apt install ocl-icd-libopencl1
sudo apt install ocl-icd-opencl-dev

I'm not sure which RPM packages to install on your version of Fedora but I did an RPM search for opencl here.

I have these 4 installed.
opencl-filesystem.noarch 1.0-9.fc30 @fedora
opencl-headers.noarch 2.2-4.20190205git49f07d3.fc30 @fedora
opencl-utils.x86_64 1-9.svn16.fc30 @fedora
opencl-utils-devel.x86_64 1-9.svn16.fc30 @fedora


I do not have

gromacs-opencl.x86_64 2019.6-1.fc30 updates
opencl-utils.i686 1-9.svn16.fc30 fedora
opencl-utils-devel.i686 1-9.svn16.fc30 fedora
openclonk.x86_64 8.1-9.20180321git570ba7a.fc30 updates
openclonk-data.noarch 8.1-9.20180321git570ba7a.fc30 updates
openclonk-devel-docs.noarch 8.1-6.20180321git570ba7a.fc30 fedora
openclonk-doc.noarch 8.1-9.20180321git570ba7a.fc30 updates

I went ahead and added the i686 rpms. Might be worth adding gromacs as well?

I'm hesitant to restart boinc though, as i'll loose all my sidock work again.

Edit
Went ahead and added gromacs as well.

Restarted boinc and took the hit to sidock. No change.

 

[biodoc]

Restarted boinc and took the hit to sidock. No change.

Could you install and run clinfo?

 

[Endgame124]

Could you install and run clinfo?

It was already installed:


clinfo
Number of platforms 2
Platform Name NVIDIA CUDA
Platform Vendor NVIDIA Corporation
Platform Version OpenCL 1.2 CUDA 10.2.159
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_nv_create_buffer cl_khr_int64_base_atomics cl_khr_int64_extended_atomics
Platform Extensions function suffix NV

Platform Name Portable Computing Language
Platform Vendor The pocl project
Platform Version OpenCL 1.2 pocl 1.3-pre RelWithDebInfo, LLVM 8.0.0, SLEEF, DISTRO, POCL_DEBUG
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd
Platform Extensions function suffix POCL

Platform Name NVIDIA CUDA
Number of devices 1
Device Name GeForce GTX 1080 Ti
Device Vendor NVIDIA Corporation
Device Vendor ID 0x10de
Device Version OpenCL 1.2 CUDA
Driver Version 440.82
Device OpenCL C Version OpenCL C 1.2
Device Type GPU
Device Topology (NV) PCI-E, 09:00.0
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 28
Max clock frequency 1582MHz
Compute Capability (NV) 6.1
Device Partition (core)
Max number of sub-devices 1
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 1024x1024x64
Max work group size 1024
Preferred work group size multiple 32
Warp size (NV) 32
Preferred / native vector sizes
char 1 / 1
short 1 / 1
int 1 / 1
long 1 / 1
half 0 / 0 (n/a)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (n/a)
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 11721506816 (10.92GiB)
Error Correction support No
Max memory allocation 2930376704 (2.729GiB)
Unified memory for Host and Device No
Integrated memory (NV) No
Minimum alignment for any data type 128 bytes
Alignment of base address 4096 bits (512 bytes)
Global Memory cache type Read/Write
Global Memory cache size 1376256 (1.312MiB)
Global Memory cache line size 128 bytes
Image support Yes
Max number of samplers per kernel 32
Max size for 1D images from buffer 268435456 pixels
Max 1D or 2D image array size 2048 images
Max 2D image size 16384x32768 pixels
Max 3D image size 16384x16384x16384 pixels
Max number of read image args 256
Max number of write image args 16
Local memory type Local
Local memory size 49152 (48KiB)
Registers per block (NV) 65536
Max number of constant args 9
Max constant buffer size 65536 (64KiB)
Max size of kernel argument 4352 (4.25KiB)
Queue properties
Out-of-order execution Yes
Profiling Yes
Prefer user sync for interop No
Profiling timer resolution 1000ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Kernel execution timeout (NV) Yes
Concurrent copy and kernel execution (NV) Yes
Number of async copy engines 2
printf() buffer size 1048576 (1024KiB)
Built-in kernels (n/a)
Device Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll cl_nv_copy_opts cl_nv_create_buffer cl_khr_int64_base_atomics cl_khr_int64_extended_atomics

Platform Name Portable Computing Language
Number of devices 1
Device Name pthread-AMD Ryzen 7 2700X Eight-Core Processor
Device Vendor AuthenticAMD
Device Vendor ID 0x6c636f70
Device Version OpenCL 1.2 pocl HSTR: pthread-x86_64-unknown-linux-gnu-znver1
Driver Version 1.3-pre
Device OpenCL C Version OpenCL C 1.2 pocl
Device Type CPU
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 16
Max clock frequency 3700MHz
Device Partition (core)
Max number of sub-devices 16
Supported partition types equally, by counts
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 4096x4096x4096
Max work group size 4096
Preferred work group size multiple 8
Preferred / native vector sizes
char 16 / 16
short 16 / 16
int 8 / 8
long 4 / 4
half 0 / 0 (n/a)
float 8 / 8
double 4 / 4 (cl_khr_fp64)
Half-precision Floating-point support (n/a)
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 31506845696 (29.34GiB)
Error Correction support No
Max memory allocation 8589934592 (8GiB)
Unified memory for Host and Device Yes
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Global Memory cache type Read/Write
Global Memory cache size 8388608 (8MiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 536870912 pixels
Max 1D or 2D image array size 2048 images
Max 2D image size 16384x16384 pixels
Max 3D image size 2048x2048x2048 pixels
Max number of read image args 128
Max number of write image args 128
Local memory type Global
Local memory size 4194304 (4MiB)
Max number of constant args 8
Max constant buffer size 4194304 (4MiB)
Max size of kernel argument 1024
Queue properties
Out-of-order execution No
Profiling Yes
Prefer user sync for interop Yes
Profiling timer resolution 1ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels Yes
printf() buffer size 16777216 (16MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_3d_image_writes cl_khr_fp64 cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64


NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV]
clCreateContext(NULL, ...) [default] Success [NV]
clCreateContext(NULL, ...) [other] Success [POCL]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) Invalid device type for platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform

ICD loader properties
ICD loader Name OpenCL ICD Loader
ICD loader Vendor OCL Icd free software
ICD loader Version 2.2.12
ICD loader Profile OpenCL 2.2

 

[biodoc]

I have these 4 installed.
opencl-filesystem.noarch 1.0-9.fc30 @fedora
opencl-headers.noarch 2.2-4.20190205git49f07d3.fc30 @fedora
opencl-utils.x86_64 1-9.svn16.fc30 @fedora
opencl-utils-devel.x86_64 1-9.svn16.fc30 @fedora

On Debian, we need the generic opencl loader and development packages from the Khronos group. For some reason the Nvidia icd loader does not work.

So search for ocl-icd and ocl-icd-dev RPM packages like this search.

The other odd thing about your clinfo output: It lists your processor as an opencl device. I don't see that on my systems.

Platform Name Portable Computing Language
Number of devices 1
Device Name pthread-AMD Ryzen 7 2700X Eight-Core Processor

 

[Endgame124]

On Debian, we need the generic opencl loader and development packages from the Khronos group. For some reason the Nvidia icd loader does not work.

So search for ocl-icd and ocl-icd-dev RPM packages like this search.

The other odd thing about your clinfo output: It lists your processor as an opencl device. I don't see that on my systems.

Platform Name Portable Computing Language
Number of devices 1
Device Name pthread-AMD Ryzen 7 2700X Eight-Core Processor

I have all available ocl-icd packages installed:

sudo dnf list installed | grep ocl-icd
ocl-icd.i686 2.2.12-3.fc30 @fedora
ocl-icd.x86_64 2.2.12-3.fc30 @fedora
ocl-icd-devel.i686 2.2.12-3.fc30 @fedora
ocl-icd-devel.x86_64 2.2.12-3.fc30 @fedora

(I'm starting to think I should just nuke fedora and either switch to centos or maybe a debian variant).

 

[Skillz]

Switch to a Debian variant. CentOS as an operating system environment is reaching EOL. They wont be supporting it anymore.

 

[biodoc]

Maybe boinc is searching in the wrong place for the appropriate libraries. For GPU computing most of us use Mint or Ubuntu on this forum. Sorry I couldn't be of more help.

 

[Endgame124]

Switch to a Debian variant. CentOS as an operating system environment is reaching EOL. They wont be supporting it anymore.

I could also switch to Oracle Linux, which might be preferable to switching entirely out of the redhat ecosystem (it is of benefit for me to be using several distros from a skill perspective. All my pis are Raspbian, so I'm trying hard to keep something in redhat).

Edit:
I think what I'm going to do after the pentathlon is move my GPU linux box to either Mint or Ubuntu (would very much love suggestions here).

My VMs that are CPU only will move to Oracle Linux.

 

[biodoc]

I have Mint Mate version 20.1 installed on my systems. The cinnamon desktop is too "busy" (steals cpu cycles from boinc apps) for me.

 

[Endgame124]

In an also extremely frustrating turn of events on getting multiple Einstein WUs to run in parallel

I set the web preferences to .5 GPU, and my main machine still doesn't recognize it - it won't run multiple WUs regardless of app_config.xml or web prefs. On this host, I do have the following under computing preference:
Using web-based preferences from:
www worldcommunitygrid com

Ended up removing Einstein and readding it. Now I'm cooking with 3 parallel processes running the GPU usually about 70% with short peaks at 80%. I could probably run 4 parallel tasks, but I'm going to let it run a little bit and get some timings before I add process #4.