ROCm 4.0.1 fails to install on Ubuntu 20.04.2

ROCm doesn't install on a fresh Ubuntu 20.04.2 installation. Errors out with the following output:
# sudo apt install rocm-dkms && sudo reboot
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  comgr dkms g++-9-multilib g++-multilib gcc-7-base gcc-8-base gcc-9-multilib
  gcc-multilib hip-base hip-doc hip-rocclr hip-samples hsa-amd-aqlprofile
  hsa-rocr-dev hsakmt-roct hsakmt-roct-dev lib32asan5 lib32atomic1
  lib32gcc-9-dev lib32gcc-s1 lib32gomp1 lib32itm1 lib32quadmath0
  lib32stdc++-9-dev lib32stdc++6 lib32ubsan1 libasan4 libc6-dev-i386
  libc6-dev-x32 libc6-i386 libc6-x32 libcilkrts5 libdrm-dev libelf-dev
  libfile-which-perl libgcc-7-dev libgl-dev libglx-dev libmpx2 libncurses5
  libpthread-stubs0-dev libstdc++-7-dev libtinfo5 libubsan0 libx11-dev
  libx32asan5 libx32atomic1 libx32gcc-9-dev libx32gcc-s1 libx32gomp1
  libx32itm1 libx32quadmath0 libx32stdc++-9-dev libx32stdc++6 libx32ubsan1
  libxau-dev libxcb1-dev libxdmcp-dev llvm-amdgpu mesa-common-dev
  openmp-extras rock-dkms rock-dkms-firmware rocm-clang-ocl rocm-cmake
  rocm-dbgapi rocm-dev rocm-device-libs rocm-gdb rocm-opencl rocm-opencl-dev
  rocm-smi rocm-smi-lib64 rocm-utils rocminfo rocprofiler-dev roctracer-dev
  x11proto-core-dev x11proto-dev xorg-sgml-doctools xtrans-dev zlib1g-dev
Suggested packages:
  menu lib32stdc++6-9-dbg libx32stdc++6-9-dbg libstdc++-7-doc libx11-doc
  libxcb-doc
The following NEW packages will be installed:
  comgr dkms g++-9-multilib g++-multilib gcc-7-base gcc-8-base gcc-9-multilib
  gcc-multilib hip-base hip-doc hip-rocclr hip-samples hsa-amd-aqlprofile
  hsa-rocr-dev hsakmt-roct hsakmt-roct-dev lib32asan5 lib32atomic1
  lib32gcc-9-dev lib32gcc-s1 lib32gomp1 lib32itm1 lib32quadmath0
  lib32stdc++-9-dev lib32stdc++6 lib32ubsan1 libasan4 libc6-dev-i386
  libc6-dev-x32 libc6-i386 libc6-x32 libcilkrts5 libdrm-dev libelf-dev
  libfile-which-perl libgcc-7-dev libgl-dev libglx-dev libmpx2 libncurses5
  libpthread-stubs0-dev libstdc++-7-dev libtinfo5 libubsan0 libx11-dev
  libx32asan5 libx32atomic1 libx32gcc-9-dev libx32gcc-s1 libx32gomp1
  libx32itm1 libx32quadmath0 libx32stdc++-9-dev libx32stdc++6 libx32ubsan1
  libxau-dev libxcb1-dev libxdmcp-dev llvm-amdgpu mesa-common-dev
  openmp-extras rock-dkms rock-dkms-firmware rocm-clang-ocl rocm-cmake
  rocm-dbgapi rocm-dev rocm-device-libs rocm-dkms rocm-gdb rocm-opencl
  rocm-opencl-dev rocm-smi rocm-smi-lib64 rocm-utils rocminfo rocprofiler-dev
  roctracer-dev x11proto-core-dev x11proto-dev xorg-sgml-doctools xtrans-dev
  zlib1g-dev
0 upgraded, 83 newly installed, 0 to remove and 0 not upgraded.
Need to get 664 MB/664 MB of archives.
After this operation, 868 MB of additional disk space will be used.
Do you want to continue?
(Reading database ... 151308 files and directories currently installed.)
Preparing to unpack .../dkms_2.8.1-5ubuntu1_all.deb ...
Unpacking dkms (2.8.1-5ubuntu1) ...
Selecting previously unselected package rock-dkms-firmware.
Preparing to unpack .../rock-dkms-firmware_1%3a4.0-23_all.deb ...
Unpacking rock-dkms-firmware (1:4.0-23) ...
Setting up rock-dkms-firmware (1:4.0-23) ...
Selecting previously unselected package rock-dkms.
(Reading database ... 151741 files and directories currently installed.)
Preparing to unpack .../00-rock-dkms_1%3a4.0-23_all.deb ...
Unpacking rock-dkms (1:4.0-23) ...
Selecting previously unselected package comgr.
Preparing to unpack .../01-comgr_<IP_ADDRESS>-rocm-rel-4.0-23-0fa438b_amd64.deb ...
Unpacking comgr (<IP_ADDRESS>-rocm-rel-4.0-23-0fa438b) ...
Selecting previously unselected package libc6-i386.
Preparing to unpack .../02-libc6-i386_2.31-0ubuntu9.2_amd64.deb ...
Unpacking libc6-i386 (2.31-0ubuntu9.2) ...
Replaced by files in installed package libc6:i386 (2.31-0ubuntu9.2) ...
Selecting previously unselected package libc6-dev-i386.
Preparing to unpack .../03-libc6-dev-i386_2.31-0ubuntu9.2_amd64.deb ...
Unpacking libc6-dev-i386 (2.31-0ubuntu9.2) ...
Selecting previously unselected package libc6-x32.
Preparing to unpack .../04-libc6-x32_2.31-0ubuntu9.2_amd64.deb ...
Unpacking libc6-x32 (2.31-0ubuntu9.2) ...
Selecting previously unselected package libc6-dev-x32.
Preparing to unpack .../05-libc6-dev-x32_2.31-0ubuntu9.2_amd64.deb ...
Unpacking libc6-dev-x32 (2.31-0ubuntu9.2) ...
Selecting previously unselected package lib32gcc-s1.
Preparing to unpack .../06-lib32gcc-s1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32gcc-s1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32gcc-s1.
Preparing to unpack .../07-libx32gcc-s1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32gcc-s1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32gomp1.
Preparing to unpack .../08-lib32gomp1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32gomp1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32gomp1.
Preparing to unpack .../09-libx32gomp1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32gomp1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32itm1.
Preparing to unpack .../10-lib32itm1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32itm1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32itm1.
Preparing to unpack .../11-libx32itm1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32itm1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32atomic1.
Preparing to unpack .../12-lib32atomic1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32atomic1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32atomic1.
Preparing to unpack .../13-libx32atomic1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32atomic1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32asan5.
Preparing to unpack .../14-lib32asan5_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking lib32asan5 (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package libx32asan5.
Preparing to unpack .../15-libx32asan5_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking libx32asan5 (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package lib32stdc++6.
Preparing to unpack .../16-lib32stdc++6_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32stdc++6 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32ubsan1.
Preparing to unpack .../17-lib32ubsan1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32ubsan1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32stdc++6.
Preparing to unpack .../18-libx32stdc++6_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32stdc++6 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32ubsan1.
Preparing to unpack .../19-libx32ubsan1_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32ubsan1 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32quadmath0.
Preparing to unpack .../20-lib32quadmath0_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking lib32quadmath0 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package libx32quadmath0.
Preparing to unpack .../21-libx32quadmath0_10.2.0-5ubuntu1~20.04_amd64.deb ...
Unpacking libx32quadmath0 (10.2.0-5ubuntu1~20.04) ...
Selecting previously unselected package lib32gcc-9-dev.
Preparing to unpack .../22-lib32gcc-9-dev_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking lib32gcc-9-dev (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package libx32gcc-9-dev.
Preparing to unpack .../23-libx32gcc-9-dev_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking libx32gcc-9-dev (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package gcc-9-multilib.
Preparing to unpack .../24-gcc-9-multilib_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking gcc-9-multilib (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package lib32stdc++-9-dev.
Preparing to unpack .../25-lib32stdc++-9-dev_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking lib32stdc++-9-dev (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package libx32stdc++-9-dev.
Preparing to unpack .../26-libx32stdc++-9-dev_9.3.0-17ubuntu1~20.04_amd64.deb ..
.
Unpacking libx32stdc++-9-dev (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package g++-9-multilib.
Preparing to unpack .../27-g++-9-multilib_9.3.0-17ubuntu1~20.04_amd64.deb ...
Unpacking g++-9-multilib (9.3.0-17ubuntu1~20.04) ...
Selecting previously unselected package gcc-multilib.
Preparing to unpack .../28-gcc-multilib_4%3a9.3.0-1ubuntu2_amd64.deb ...
Unpacking gcc-multilib (4:9.3.0-1ubuntu2) ...
Selecting previously unselected package g++-multilib.
Preparing to unpack .../29-g++-multilib_4%3a9.3.0-1ubuntu2_amd64.deb ...
Unpacking g++-multilib (4:9.3.0-1ubuntu2) ...
Selecting previously unselected package gcc-7-base:amd64.
Preparing to unpack .../30-gcc-7-base_7.5.0-6ubuntu2_amd64.deb ...
Unpacking gcc-7-base:amd64 (7.5.0-6ubuntu2) ...
Selecting previously unselected package gcc-8-base:amd64.
Preparing to unpack .../31-gcc-8-base_8.4.0-3ubuntu2_amd64.deb ...
Unpacking gcc-8-base:amd64 (8.4.0-3ubuntu2) ...
Selecting previously unselected package libfile-which-perl.
Preparing to unpack .../32-libfile-which-perl_1.23-1_all.deb ...
Unpacking libfile-which-perl (1.23-1) ...
Selecting previously unselected package hip-base.
Preparing to unpack .../33-hip-base_4.0.20496.5685.40000-23_amd64.deb ...
Unpacking hip-base (4.0.20496.5685.40000-23) ...
Selecting previously unselected package hip-doc.
Preparing to unpack .../34-hip-doc_4.0.20496.5685.40000-23_amd64.deb ...
Unpacking hip-doc (4.0.20496.5685.40000-23) ...
Selecting previously unselected package hsakmt-roct.
Preparing to unpack .../35-hsakmt-roct_20201016.1.0269-mainline-20201016-1-g0269
ce3_amd64.deb ...
Unpacking hsakmt-roct (20201016.1.0269-mainline-20201016-1-g0269ce3) ...
Selecting previously unselected package hsa-rocr-dev.
Preparing to unpack .../36-hsa-rocr-dev_1.2.40000.0-rocm-rel-4.0-23-a5173c90_amd
64.deb ...
Unpacking hsa-rocr-dev (1.2.40000.0-rocm-rel-4.0-23-a5173c90) ...
Selecting previously unselected package rocminfo.
Preparing to unpack .../37-rocminfo_1.40000.0_amd64.deb ...
Unpacking rocminfo (1.40000.0) ...
Selecting previously unselected package libasan4:amd64.
Preparing to unpack .../38-libasan4_7.5.0-6ubuntu2_amd64.deb ...
Unpacking libasan4:amd64 (7.5.0-6ubuntu2) ...
Selecting previously unselected package libubsan0:amd64.
Preparing to unpack .../39-libubsan0_7.5.0-6ubuntu2_amd64.deb ...
Unpacking libubsan0:amd64 (7.5.0-6ubuntu2) ...
Selecting previously unselected package libcilkrts5:amd64.
Preparing to unpack .../40-libcilkrts5_7.5.0-6ubuntu2_amd64.deb ...
Unpacking libcilkrts5:amd64 (7.5.0-6ubuntu2) ...
Selecting previously unselected package libmpx2:amd64.
Preparing to unpack .../41-libmpx2_8.4.0-3ubuntu2_amd64.deb ...
Unpacking libmpx2:amd64 (8.4.0-3ubuntu2) ...
Selecting previously unselected package libgcc-7-dev:amd64.
Preparing to unpack .../42-libgcc-7-dev_7.5.0-6ubuntu2_amd64.deb ...
Unpacking libgcc-7-dev:amd64 (7.5.0-6ubuntu2) ...
Selecting previously unselected package libstdc++-7-dev:amd64.
Preparing to unpack .../43-libstdc++-7-dev_7.5.0-6ubuntu2_amd64.deb ...
Unpacking libstdc++-7-dev:amd64 (7.5.0-6ubuntu2) ...
Selecting previously unselected package llvm-amdgpu.
Preparing to unpack .../44-llvm-amdgpu_12.0.dev_amd64.deb ...
Unpacking llvm-amdgpu (12.0.dev) ...
Selecting previously unselected package hip-rocclr.
Preparing to unpack .../45-hip-rocclr_4.0.20496.5685.40000-23_amd64.deb ...
Unpacking hip-rocclr (4.0.20496.5685.40000-23) ...
Selecting previously unselected package hip-samples.
Preparing to unpack .../46-hip-samples_4.0.20496.5685.40000-23_amd64.deb ...
Unpacking hip-samples (4.0.20496.5685.40000-23) ...
Selecting previously unselected package hsa-amd-aqlprofile.
Preparing to unpack .../47-hsa-amd-aqlprofile_1.0.0_amd64.deb ...
Unpacking hsa-amd-aqlprofile (1.0.0) ...
Selecting previously unselected package hsakmt-roct-dev.
Preparing to unpack .../48-hsakmt-roct-dev_20201016.1.0269-mainline-20201016-1-g
0269ce3_amd64.deb ...
Unpacking hsakmt-roct-dev (20201016.1.0269-mainline-20201016-1-g0269ce3) ...
Selecting previously unselected package libdrm-dev:amd64.
Preparing to unpack .../49-libdrm-dev_2.4.102-1ubuntu1~20.04.1_amd64.deb ...
Unpacking libdrm-dev:amd64 (2.4.102-1ubuntu1~20.04.1) ...
Selecting previously unselected package zlib1g-dev:amd64.
Preparing to unpack .../50-zlib1g-dev_1%3a1.2.11.dfsg-2ubuntu1.2_amd64.deb ...
Unpacking zlib1g-dev:amd64 (1:1.2.11.dfsg-2ubuntu1.2) ...
Selecting previously unselected package libelf-dev:amd64.
Preparing to unpack .../51-libelf-dev_0.176-1.1build1_amd64.deb ...
Unpacking libelf-dev:amd64 (0.176-1.1build1) ...
Selecting previously unselected package xorg-sgml-doctools.
Preparing to unpack .../52-xorg-sgml-doctools_1%3a1.11-1_all.deb ...
Unpacking xorg-sgml-doctools (1:1.11-1) ...
Selecting previously unselected package x11proto-dev.
Preparing to unpack .../53-x11proto-dev_2019.2-1ubuntu1_all.deb ...
Unpacking x11proto-dev (2019.2-1ubuntu1) ...
Selecting previously unselected package x11proto-core-dev.
Preparing to unpack .../54-x11proto-core-dev_2019.2-1ubuntu1_all.deb ...
Unpacking x11proto-core-dev (2019.2-1ubuntu1) ...
Selecting previously unselected package libxau-dev:amd64.
Preparing to unpack .../55-libxau-dev_1%3a1.0.9-0ubuntu1_amd64.deb ...
Unpacking libxau-dev:amd64 (1:1.0.9-0ubuntu1) ...
Selecting previously unselected package libxdmcp-dev:amd64.
Preparing to unpack .../56-libxdmcp-dev_1%3a1.1.3-0ubuntu1_amd64.deb ...
Unpacking libxdmcp-dev:amd64 (1:1.1.3-0ubuntu1) ...
Selecting previously unselected package xtrans-dev.
Preparing to unpack .../57-xtrans-dev_1.4.0-1_all.deb ...
Unpacking xtrans-dev (1.4.0-1) ...
Selecting previously unselected package libpthread-stubs0-dev:amd64.
Preparing to unpack .../58-libpthread-stubs0-dev_0.4-1_amd64.deb ...
Unpacking libpthread-stubs0-dev:amd64 (0.4-1) ...
Selecting previously unselected package libxcb1-dev:amd64.
Preparing to unpack .../59-libxcb1-dev_1.14-2_amd64.deb ...
Unpacking libxcb1-dev:amd64 (1.14-2) ...
Selecting previously unselected package libx11-dev:amd64.
Preparing to unpack .../60-libx11-dev_2%3a1.6.9-2ubuntu1.1_amd64.deb ...
Unpacking libx11-dev:amd64 (2:1.6.9-2ubuntu1.1) ...
Selecting previously unselected package libglx-dev:amd64.
Preparing to unpack .../61-libglx-dev_1.3.2-1~ubuntu<IP_ADDRESS>_amd64.deb ...
Unpacking libglx-dev:amd64 (1.3.2-1~ubuntu<IP_ADDRESS>) ...
Selecting previously unselected package libgl-dev:amd64.
Preparing to unpack .../62-libgl-dev_1.3.2-1~ubuntu<IP_ADDRESS>_amd64.deb ...
Unpacking libgl-dev:amd64 (1.3.2-1~ubuntu<IP_ADDRESS>) ...
Selecting previously unselected package libtinfo5:amd64.
Preparing to unpack .../63-libtinfo5_6.2-0ubuntu2_amd64.deb ...
Unpacking libtinfo5:amd64 (6.2-0ubuntu2) ...
Selecting previously unselected package libncurses5:amd64.
Preparing to unpack .../64-libncurses5_6.2-0ubuntu2_amd64.deb ...
Unpacking libncurses5:amd64 (6.2-0ubuntu2) ...
Selecting previously unselected package mesa-common-dev:amd64.
Preparing to unpack .../65-mesa-common-dev_20.2.6-0ubuntu<IP_ADDRESS>_amd64.deb ...
Unpacking mesa-common-dev:amd64 (20.2.6-0ubuntu<IP_ADDRESS>) ...
Selecting previously unselected package rocm-device-libs.
Preparing to unpack .../66-rocm-device-libs_<IP_ADDRESS>7-rocm-rel-4.0-23-db8c0c3_am
d64.deb ...
Unpacking rocm-device-libs (<IP_ADDRESS>7-rocm-rel-4.0-23-db8c0c3) ...
Selecting previously unselected package openmp-extras.
Preparing to unpack .../67-openmp-extras_12.10-0_amd64.deb ...
Unpacking openmp-extras (12.10-0) ...
Selecting previously unselected package rocm-opencl.
Preparing to unpack .../68-rocm-opencl_3.6Beta-17-g875c1f8-rocm-rel-4.0-23_amd64
.deb ...
Unpacking rocm-opencl (3.6Beta-17-g875c1f8-rocm-rel-4.0-23) ...
Selecting previously unselected package rocm-opencl-dev.
Preparing to unpack .../69-rocm-opencl-dev_3.6Beta-17-g875c1f8-rocm-rel-4.0-23_a
md64.deb ...
Unpacking rocm-opencl-dev (3.6Beta-17-g875c1f8-rocm-rel-4.0-23) ...
Selecting previously unselected package rocm-clang-ocl.
Preparing to unpack .../70-rocm-clang-ocl_<IP_ADDRESS>-rocm-rel-4.0-23-50fb51a_amd64
.deb ...
Unpacking rocm-clang-ocl (<IP_ADDRESS>-rocm-rel-4.0-23-50fb51a) ...
Selecting previously unselected package rocm-cmake.
Preparing to unpack .../71-rocm-cmake_<IP_ADDRESS>-rocm-rel-4.0-23-1d1caa5_amd64.de
b ...
Unpacking rocm-cmake (<IP_ADDRESS>-rocm-rel-4.0-23-1d1caa5) ...
Selecting previously unselected package rocm-dbgapi.
Preparing to unpack .../72-rocm-dbgapi_<IP_ADDRESS>000-23_amd64.deb ...
Unpacking rocm-dbgapi (<IP_ADDRESS>000-23) ...
Selecting previously unselected package rocm-gdb.
Preparing to unpack .../73-rocm-gdb_10.1-rocm-rel-4.0-23_amd64.deb ...
Unpacking rocm-gdb (10.1-rocm-rel-4.0-23) ...
Selecting previously unselected package rocm-smi-lib64.
Preparing to unpack .../74-rocm-smi-lib64_<IP_ADDRESS>-rocm-rel-4.0-23-4b49d2d_amd64.
deb ...
Unpacking rocm-smi-lib64 (<IP_ADDRESS>-rocm-rel-4.0-23-4b49d2d) ...
Selecting previously unselected package rocm-smi.
Preparing to unpack .../75-rocm-smi_1.0.0-206-rocm-rel-4.0-23-ge39c0e2_amd64.deb
 ...
Unpacking rocm-smi (1.0.0-206-rocm-rel-4.0-23-ge39c0e2) ...
Selecting previously unselected package rocm-utils.
Preparing to unpack .../76-rocm-utils_<IP_ADDRESS>000-23_amd64.deb ...
Unpacking rocm-utils (<IP_ADDRESS>000-23) ...
Selecting previously unselected package rocprofiler-dev.
Preparing to unpack .../77-rocprofiler-dev_1.0.0_amd64.deb ...
Unpacking rocprofiler-dev (1.0.0) ...
Selecting previously unselected package roctracer-dev.
Preparing to unpack .../78-roctracer-dev_1.0.0_amd64.deb ...
Unpacking roctracer-dev (1.0.0) ...
Selecting previously unselected package rocm-dev.
Preparing to unpack .../79-rocm-dev_<IP_ADDRESS>000-23_amd64.deb ...
Unpacking rocm-dev (<IP_ADDRESS>000-23) ...
Selecting previously unselected package rocm-dkms.
Preparing to unpack .../80-rocm-dkms_<IP_ADDRESS>000-23_amd64.deb ...
Unpacking rocm-dkms (<IP_ADDRESS>000-23) ...
Setting up comgr (<IP_ADDRESS>-rocm-rel-4.0-23-0fa438b) ...
Setting up libfile-which-perl (1.23-1) ...
Setting up libdrm-dev:amd64 (2.4.102-1ubuntu1~20.04.1) ...
Setting up hip-base (4.0.20496.5685.40000-23) ...
Setting up rocm-cmake (<IP_ADDRESS>-rocm-rel-4.0-23-1d1caa5) ...
Setting up roctracer-dev (1.0.0) ...
Setting up rocminfo (1.40000.0) ...
Setting up hsa-amd-aqlprofile (1.0.0) ...
Setting up libpthread-stubs0-dev:amd64 (0.4-1) ...
Setting up dkms (2.8.1-5ubuntu1) ...
Setting up rocm-smi-lib64 (<IP_ADDRESS>-rocm-rel-4.0-23-4b49d2d) ...
Setting up libc6-x32 (2.31-0ubuntu9.2) ...
Setting up rocm-device-libs (<IP_ADDRESS>7-rocm-rel-4.0-23-db8c0c3) ...
Setting up rock-dkms (1:4.0-23) ...
Loading new amdgpu-4.0-23 DKMS files...
Building for 5.8.0-41-generic
Building for architecture x86_64
Building initial module for 5.8.0-41-generic
Error! Bad return status for module build on kernel: 5.8.0-41-generic (x86_64)
Consult /var/lib/dkms/amdgpu/4.0-23/build/make.log for more information.
dpkg: error processing package rock-dkms (--configure):
 installed rock-dkms package post-installation script subprocess returned error
exit status 10
Setting up rocm-smi (1.0.0-206-rocm-rel-4.0-23-ge39c0e2) ...
Setting up xtrans-dev (1.4.0-1) ...
Setting up gcc-7-base:amd64 (7.5.0-6ubuntu2) ...
Setting up libasan4:amd64 (7.5.0-6ubuntu2) ...
Setting up rocm-dbgapi (<IP_ADDRESS>000-23) ...
Setting up gcc-8-base:amd64 (8.4.0-3ubuntu2) ...
Setting up libx32gomp1 (10.2.0-5ubuntu1~20.04) ...
Setting up libmpx2:amd64 (8.4.0-3ubuntu2) ...
Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-2ubuntu1.2) ...
Setting up libcilkrts5:amd64 (7.5.0-6ubuntu2) ...
Setting up libubsan0:amd64 (7.5.0-6ubuntu2) ...
Setting up libc6-i386 (2.31-0ubuntu9.2) ...
Setting up libx32quadmath0 (10.2.0-5ubuntu1~20.04) ...
Setting up xorg-sgml-doctools (1:1.11-1) ...
dpkg: dependency problems prevent configuration of rocm-dkms:
 rocm-dkms depends on rock-dkms; however:
  Package rock-dkms is not configured yet.

dpkg: error processing package rocm-dkms (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup erro
r from a previous failure.
                          Setting up hip-doc (4.0.20496.5685.40000-23) ...
Setting up hsakmt-roct (20201016.1.0269-mainline-20201016-1-g0269ce3) ...
Setting up libtinfo5:amd64 (6.2-0ubuntu2) ...
Setting up lib32atomic1 (10.2.0-5ubuntu1~20.04) ...
Setting up libx32atomic1 (10.2.0-5ubuntu1~20.04) ...
Setting up libc6-dev-i386 (2.31-0ubuntu9.2) ...
Setting up lib32itm1 (10.2.0-5ubuntu1~20.04) ...
Setting up hip-samples (4.0.20496.5685.40000-23) ...
Setting up libx32gcc-s1 (10.2.0-5ubuntu1~20.04) ...
Setting up libx32itm1 (10.2.0-5ubuntu1~20.04) ...
Setting up hsa-rocr-dev (1.2.40000.0-rocm-rel-4.0-23-a5173c90) ...
Setting up libncurses5:amd64 (6.2-0ubuntu2) ...
Setting up libc6-dev-x32 (2.31-0ubuntu9.2) ...
Setting up rocm-gdb (10.1-rocm-rel-4.0-23) ...
Setting up lib32gomp1 (10.2.0-5ubuntu1~20.04) ...
Setting up lib32gcc-s1 (10.2.0-5ubuntu1~20.04) ...
Setting up lib32stdc++6 (10.2.0-5ubuntu1~20.04) ...
Setting up libelf-dev:amd64 (0.176-1.1build1) ...
Setting up rocm-opencl (3.6Beta-17-g875c1f8-rocm-rel-4.0-23) ...
Setting up libgcc-7-dev:amd64 (7.5.0-6ubuntu2) ...
Setting up libx32asan5 (9.3.0-17ubuntu1~20.04) ...
Setting up lib32quadmath0 (10.2.0-5ubuntu1~20.04) ...
Setting up hsakmt-roct-dev (20201016.1.0269-mainline-20201016-1-g0269ce3) ...
Setting up lib32asan5 (9.3.0-17ubuntu1~20.04) ...
Setting up libstdc++-7-dev:amd64 (7.5.0-6ubuntu2) ...
Setting up libx32stdc++6 (10.2.0-5ubuntu1~20.04) ...
Setting up libx32ubsan1 (10.2.0-5ubuntu1~20.04) ...
Setting up rocprofiler-dev (1.0.0) ...
Setting up lib32ubsan1 (10.2.0-5ubuntu1~20.04) ...
Setting up llvm-amdgpu (12.0.dev) ...
Setting up lib32gcc-9-dev (9.3.0-17ubuntu1~20.04) ...
Setting up libx32gcc-9-dev (9.3.0-17ubuntu1~20.04) ...
Setting up lib32stdc++-9-dev (9.3.0-17ubuntu1~20.04) ...
Setting up libx32stdc++-9-dev (9.3.0-17ubuntu1~20.04) ...
Setting up openmp-extras (12.10-0) ...
Setting up gcc-9-multilib (9.3.0-17ubuntu1~20.04) ...
Setting up hip-rocclr (4.0.20496.5685.40000-23) ...
Setting up gcc-multilib (4:9.3.0-1ubuntu2) ...
Setting up g++-9-multilib (9.3.0-17ubuntu1~20.04) ...
Setting up g++-multilib (4:9.3.0-1ubuntu2) ...
Processing triggers for sgml-base (1.29.1) ...
Setting up x11proto-dev (2019.2-1ubuntu1) ...
Setting up libxau-dev:amd64 (1:1.0.9-0ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for man-db (2.9.1-1) ...
Setting up libxdmcp-dev:amd64 (1:1.1.3-0ubuntu1) ...
Setting up x11proto-core-dev (2019.2-1ubuntu1) ...
Setting up libxcb1-dev:amd64 (1.14-2) ...
Setting up libx11-dev:amd64 (2:1.6.9-2ubuntu1.1) ...
Setting up libglx-dev:amd64 (1.3.2-1~ubuntu<IP_ADDRESS>) ...
Setting up libgl-dev:amd64 (1.3.2-1~ubuntu<IP_ADDRESS>) ...
Setting up mesa-common-dev:amd64 (20.2.6-0ubuntu<IP_ADDRESS>) ...
Setting up rocm-opencl-dev (3.6Beta-17-g875c1f8-rocm-rel-4.0-23) ...
Setting up rocm-clang-ocl (<IP_ADDRESS>-rocm-rel-4.0-23-50fb51a) ...
Setting up rocm-utils (<IP_ADDRESS>000-23) ...
Setting up rocm-dev (<IP_ADDRESS>000-23) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Errors were encountered while processing:
 rock-dkms
 rocm-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)

neofetch:
OS: Ubuntu 20.04.2 LTS x86_64
Kernel: 5.8.0-41-generic
Uptime: 1 hour, 15 mins
Packages: 1782 (dpkg), 6 (snap)
Shell: bash 5.0.17
Resolution: 1920x1200
DE: GNOME
WM: Mutter
WM Theme: Adwaita
Theme: Yaru [GTK2/3]
Icons: Yaru [GTK2/3]
Terminal: gnome-terminal
CPU: AMD Ryzen Threadripper 1900X (16) @ 3.800GHz
GPU: AMD ATI Radeon RX Vega 56/64
Memory: 2104MiB / 15931MiB

This is a kernel-5.8 issue. ROCm-4.0 cannot support kernel-5.8.
https://github.com/RadeonOpenCompute/ROCm/issues/1367
https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/issues/107
ROCm-4.1 maybe support linux-5.8.
Recently, we should install linux-5.4 and remove linux-5.8, before installing ROCm-4.0

Thanks for responding so quickly.
You might want to pin this issue as there will probably be a lot of new duplicates otherwise.

Try changing your kernel to 5.6 oem. It worked for me. However, after installing on the 5.6 kernel rocm is unable to detect my external gpu. Not sure if you would face the same problem.
After installing you could add yourself to the video groups which you can find in the official amd guide page to not face an issue of unable to detecting the gpu.

Hi @KristijanZic
Thanks for reaching us.
As @xuhuisheng informed, ROCm currently does not work on 5.8 kernel.
We are coming up with 5.8 kernel support from ROCm 4.1, its on the way. Please stay tuned.

I had to go back to the GA kernel on 20.04 and then it passed installation.

I had to go back to the GA kernel on 20.04 and then it passed installation.

Thanks, its the case now. We are coming up with 5.8 kernel support from ROCm 4.1.


Gradle build changes: version upgrades for gradle, plugins and dependencies

Details:

gradle: 6.8.2 -->> 6.9
'edu.sc.seis.launch4j' plugin: 2.4.9 -->> 2.5.0
'org.hsqldb:hsqldb' 2.5.1 -->> 2.6.0
'com.google.protobuf:protobuf-java' 3.14.0 -->> 3.16.0
'junit:junit' 4.13.1 -->> 4.13.2

Next time test this properly!

Mea culpa @onyxbits !
Pardon me for asking (and I am not being sarcastic here): how can we execute tests in a proper way ?
Thing is that ./gradlew clean test returns inconclusive results (at least in my case, see attached snapshots: gradle tests execution stuck on 87% on both Windows and Linux).
I guess I should created draft PR instead and then clarify with you why tests are inconclusive, but it is what it is. I had a good run with 2 previous commits being successful, but all good things come to an end.
Assumable I should that generate .exe binary, start it and download some apk; and, no I didn't do that (guilty as charged).
Speaking of testing: what do you think about adding CI server test executions for push requests ? I would be interested to help with that: https://github.com/features/actions
Let me know if you interested, I have a rich experience with CI/CD.

Do NOT up lib versions simply because there's a new release of something available. In particular do NOT send pull requests without explaining why a new version of lib xyz is needed.
As per your comment above: https://github.com/onyxbits/raccoon4/pull/93#issuecomment-834723828 this is just an upgrade for a sake of upgrade, nothing more and nothing less.

* The HSQL update uses a new class version which is incompatible with the project

Correct. Thing is that HSQL release notes are hosted on their site and code is on SourceForge. Now I realize that I should bump HSQL version to a 2.5.2 and not to 2.6.0.
Version 2.6.0

20th Anniversary Release

March 2021. Version 2.6.0 of HyperSQL database management system adds support for Java module system and enhancements in many areas. Zip package at the download link above contains Java 11 and 8 jars. Individual jars for Java 6, 8 and 11 are available at http://hsqldb.org/download. Jars are also available at Maven repositories as version 2.6.0 for Java 11 and version 2.5.2 for Java 8.

* Something (couldn't figure out what exactly) resulted in the JAR file not getting a Main-class set

Maybe is due to a gradle plugin upgrade; breaking changes are not mentioned, so I will have to dig deeper (just by the looks of it it seems that only thing needs to be changed after bumping to 2.5.0 is this line: jar = "lib/${project.tasks.fatJar.archiveName}"

https://github.com/TheBoegl/gradle-launch4j/blob/v2.5.0/VERSION.md
git diff between gradle-launch4 2.4.9 and 2.5.0

One more thing: I am not bashing gradle-launch4j creator here.... I should checked this git diff before I decided to bump this plugin version.
So there you have it @onyxbits... As I stated above I could try to help with CI server test execution for PRs and I hope you will accept my extended hand.
Test snapshots (as mentioned above):

@dejan2609
First of all: sorry for snapping at you. I was in a hurry, carelessly merged the PR and then got pissed because stuff broke and I didn't have the time for that. So, partly my fault.

Pardon me for asking (and I am not being sarcastic here): how can we execute tests in a proper way ?
[..]
Assumable I should that generate .exe binary, start it and download some apk; and, no I didn't do that (guilty as charged).

The master branch must always build, so yes, creating the binary would have been the minimal test to do here.

Speaking of testing: what do you think about adding CI server test executions for push requests ? I would be interested to help with that: https://github.com/features/actions

I'm quite interested in adding this, though in its current state, the source is too tightly coupled with the UI to make this meaningful.

Maybe is due to a gradle plugin upgrade; breaking changes are not mentioned, so I will have to dig deeper

Let's just leave it at the current version. It works and there are no security concerns at this point.

@onyxbits I agree, from a build POV there are no reasons to force any changes at the moment.
At some point of time (and only after gradle-launch4j plugin gets  Gradle version 7 upgrade) I could create some draft PR... but lets leave that to the future.


User nou -> create organization: claim-ul de organizationid nu se actualizeaza la salvare

Migrated from https://github.com/gov-ithub/auth-sso/issues/91
Originally created by @alexchiraples on Tue, 03 Jan 2017 18:23:19 GMT

Ce se intampla acum e ca raman blocat in fereastra de adauga organizatie

Migrated from https://github.com/gov-ithub/auth-sso/issues/91#issuecomment-270203783
Originally created by @dmihaita on Tue, 03 Jan 2017 19:39:01 GMT

Problema e rezolvata in branch iss50-org-select . Maine fac un wrap-up la acel branch si o sa fac un pull request.


mnist standalone spark on training "ValueError: When using replicas, all Variables must have their device set: name: "hid_w"

Hello, having trouble getting the training portion of the example to work on tensorflow 1.4
Similar to the following issue on Stackoverflow https://stackoverflow.com/questions/38793718/distributed-tensorflow-valueerror-when-when-using-replicas-all-variables-mus
"CentOS Linux release 7.3.1611 (Core) "
[root@I77700 ~]# uname -a
Linux I77700 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@I77700 ~]#
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/04 09:34:23 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
17/11/04 09:34:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform...
.
.
.
17/11/04 09:34:27 INFO HadoopRDD: Input split: file:/root/Downloads/TensorFlowOnSpark-master/examples/mnist/csv/train/images/part-00001:0+11231804
17/11/04 09:34:27 INFO TorrentBroadcast: Started reading broadcast variable 0
17/11/04 09:34:27 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.3 KB, free 56.1 KB)
17/11/04 09:34:27 INFO TorrentBroadcast: Reading broadcast variable 0 took 8 ms
17/11/04 09:34:27 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 280.8 KB, free 336.9 KB)
17/11/04 09:34:27 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/11/04 09:34:27 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/11/04 09:34:27 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/11/04 09:34:27 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/11/04 09:34:27 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/11/04 09:34:27 INFO HadoopRDD: Input split: file:/root/Downloads/TensorFlowOnSpark-master/examples/mnist/csv/train/labels/part-00001:0+245760
17/11/04 09:34:27 INFO TorrentBroadcast: Started reading broadcast variable 1
17/11/04 09:34:27 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 19.3 KB, free 356.2 KB)
17/11/04 09:34:27 INFO TorrentBroadcast: Reading broadcast variable 1 took 9 ms
17/11/04 09:34:27 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 280.8 KB, free 637.0 KB)
2017-11-04 09:34:27,322 INFO (MainThread-4406) Connected to TFSparkNode.mgr on <IP_ADDRESS>, ppid=4314, state='running'
2017-11-04 09:34:27,325 INFO (MainThread-4406) mgr.state='running'
2017-11-04 09:34:27,325 INFO (MainThread-4406) Feeding partition <generator object load_stream at 0x2c84870> into input queue <multiprocessing.queues.JoinableQueue object at 0x2bc39d0>
17/11/04 09:34:28 INFO PythonRunner: Times: total = 1838, boot = -198, init = 204, finish = 1832
17/11/04 09:34:28 INFO PythonRunner: Times: total = 63, boot = 5, init = 8, finish = 50
2017-11-04 09:34:29,505 INFO (MainThread-4396) 1: ======== worker:1 ========
2017-11-04 09:34:29,505 INFO (MainThread-4396) 1: Cluster spec: {'worker': ['<IP_ADDRESS>:40398', '<IP_ADDRESS>:41647']}
2017-11-04 09:34:29,505 INFO (MainThread-4396) 1: Using CPU
2017-11-04 09:34:29.505697: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
E1104 09:34:29.506359927    4396 ev_epoll1_linux.c:1051]     grpc epoll fd: 12
2017-11-04 09:34:29.507902: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> <IP_ADDRESS>:40398, 1 -> localhost:41647}
2017-11-04 09:34:29.508290: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:41647
tensorflow model path: file:///root/Downloads/TensorFlowOnSpark-master/mnist_model
Process Process-2:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "mnist_dist.py", line 121, in map_fun
save_model_secs=10)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 336, in init
self._verify_setup()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 885, in _verify_setup
"their device set: %s" % op)
ValueError: When using replicas, all Variables must have their device set: name: "hid_w"
op: "VariableV2"
attr {
key: "container"
value {
s: ""
}
}
attr {
key: "dtype"
value {
type: DT_FLOAT
}
}
attr {
key: "shape"
value {
shape {
dim {
size: 784
}
dim {
size: 128
}
}
}
}
attr {
key: "shared_name"
value {
s: ""
}
}
Thanks.
Same results for 1.3
tensorflow (1.3.0)
tensorflow-tensorboard (0.1.8)
tensorflowonspark (1.0.7)

mnist_deep.py CPU:
[root@I77700 ~]# python mnist_deep.py
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
2017-11-04 13:55:41.526736: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 13:55:41.526769: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 13:55:41.526776: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 13:55:41.526781: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 13:55:41.526787: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
step 0, training accuracy 0.02
step 100, training accuracy 0.9
step 200, training accuracy 0.88
step 300, training accuracy 0.98
step 400, training accuracy 0.98
step 500, training accuracy 0.96
step 600, training accuracy 0.96
step 700, training accuracy 0.94
step 800, training accuracy 0.96
step 900, training accuracy 0.94
test accuracy 0.9621
[root@I77700 ~]#
switch to tensorflow-gpu:
[root@I77700 ~]# pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.3.0-cp27-none-linux_x86_64.whl
[root@I77700 ~]# python mnist_deep6.py
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
2017-11-04 14:03:44.841218: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 14:03:44.841246: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 14:03:44.841253: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 14:03:44.841258: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 14:03:44.841264: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-11-04 14:03:45.213720: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-11-04 14:03:45.214020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1050
major: 6 minor: 1 memoryClockRate (GHz) 1.455
pciBusID 0000:01:00.0
Total memory: 1.95GiB
Free memory: 1.79GiB
2017-11-04 14:03:45.214075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2017-11-04 14:03:45.214086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y
2017-11-04 14:03:45.214101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0)
step 0, training accuracy 0.16
step 100, training accuracy 0.78
step 200, training accuracy 0.92
.
.
.
step 19800, training accuracy 1
step 19900, training accuracy 1
2017-11-04 14:07:04.765106: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 523.38MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-11-04 14:07:04.765161: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.82GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-11-04 14:07:04.765198: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 958.90MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-11-04 14:07:05.121030: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.63GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-11-04 14:07:05.121066: W tensorflow/core/common_runtime/bfc_allocator.cc:217] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.73GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
test accuracy 0.991429
[root@I77700 ~]#
[root@I77700 ~]# nvidia-smi
Sat Nov  4 14:04:31 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1050    Off  | 0000:01:00.0      On |                  N/A |
| 35%   39C    P0    57W /  75W |   1945MiB /  1995MiB |     74%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      2539    G   /usr/bin/Xorg                                   80MiB |
|    0      3114    G   /usr/bin/gnome-shell                            48MiB |
|    0      6954    C   python                                        1813MiB |
+-----------------------------------------------------------------------------+
[root@I77700 ~]# nvidia-smi --query-gpu=timestamp,utilization.gpu,utilization.memory,temperature.gpu,memory.total,memory.free,pstate --format=csv -l 1
timestamp, utilization.gpu [%], utilization.memory [%], temperature.gpu, memory.total [MiB], memory.free [MiB], pstate
2017/11/04 14:04:40.508, 80 %, 52 %, 40, 1995 MiB, 50 MiB, P0
2017/11/04 14:04:41.509, 78 %, 50 %, 40, 1995 MiB, 50 MiB, P0
2017/11/04 14:04:42.509, 77 %, 50 %, 40, 1995 MiB, 50 MiB, P0
2017/11/04 14:04:43.510, 73 %, 47 %, 40, 1995 MiB, 50 MiB, P0
2017/11/04 14:04:44.510, 82 %, 53 %, 40, 1995 MiB, 50 MiB, P0
2017/11/04 14:04:45.511, 84 %, 54 %, 40, 1995 MiB, 51 MiB, P0
2017/11/04 14:04:46.511, 78 %, 50 %, 40, 1995 MiB, 53 MiB, P0
2017/11/04 14:04:47.512, 81 %, 53 %, 40, 1995 MiB, 53 MiB, P0
2017/11/04 14:04:48.512, 86 %, 55 %, 41, 1995 MiB, 53 MiB, P0
2017/11/04 14:04:49.512, 72 %, 47 %, 40, 1995 MiB, 53 MiB, P0
2017/11/04 14:04:50.513, 78 %, 50 %, 41, 1995 MiB, 53 MiB, P0
^C[root@I77700 ~]#

mnist_deep6.py updates to workaround OOM
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(20000):
batch = mnist.train.next_batch(50)
if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict={
x: batch[0], y_: batch[1], keep_prob: 1.0})
print('step %d, training accuracy %g' % (i, train_accuracy))
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
test_image = mnist.test.images[0:7000, :]
test_label = mnist.test.labels[0:7000, :]
print("test accuracy %g"%accuracy.eval(feed_dict={x: test_image, y_: test_label, keep_prob: 1.0}))

Can I assume that the original error When using replicas, all Variables must have their device set: name: "hid_w" is now fixed? (since you're seeing training accuracy numbers and generating profiling output).

Hello, thanks for getting back to me.  The error is still there, the training accuracy numbers are from standalone tensorflow, I figured I would make sure this worked also, thanks.

I see.  Can you send me your full logs of a failed TFoS run (and your command-line)?  (You can post here or send directly to me: leewyang@gmail.com).  Unfortunately, I haven't seen this particular issue on this example before... so I'm not sure what's going on...

You bet, I'll get started, standalone spark first.  I've attached the console output from loading the csv files.

tfonspark.standalone.loadcsv.txt

Works just fine with one worker, thanks a ton, I'll close this issue and check out using HDP2.6.3, thanks again.
[root@I77700 TensorFlowOnSpark-master]# export SPARK_HOME=$(pwd)/spark-1.6.0-bin-hadoop2.6
[root@I77700 TensorFlowOnSpark-master]# export PATH=${SPARK_HOME}/bin:${PATH}
[root@I77700 TensorFlowOnSpark-master]# export MASTER=spark://I77700:7077
[root@I77700 TensorFlowOnSpark-master]# export SPARK_WORKER_INSTANCES=1
[root@I77700 TensorFlowOnSpark-master]# export CORES_PER_WORKER=1
[root@I77700 TensorFlowOnSpark-master]# export TOTAL_CORES=$((${CORES_PER_WORKER}+${SPARK_WORKER_INSTANCES}))
[root@I77700 TensorFlowOnSpark-master]# ${SPARK_HOME}/sbin/start-master.sh; ${SPARK_HOME}/sbin/start-slave.sh -c $CORES_PER_WORKER -m 3G ${MASTER}
starting org.apache.spark.deploy.master.Master, logging to /root/Downloads/TensorFlowOnSpark-master/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-I77700.out
starting org.apache.spark.deploy.worker.Worker, logging to /root/Downloads/TensorFlowOnSpark-master/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-I77700.out
[root@I77700 TensorFlowOnSpark-master]# ls -l mnist_model
total 0
[root@I77700 TensorFlowOnSpark-master]# rm -rf mnist_model
[root@I77700 TensorFlowOnSpark-master]# ${SPARK_HOME}/bin/spark-submit \

--master ${MASTER}
--py-files ${TFoS_HOME}/examples/mnist/spark/mnist_dist.py
--conf spark.cores.max=${TOTAL_CORES}
--conf spark.task.cpus=${CORES_PER_WORKER}
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME"
${TFoS_HOME}/examples/mnist/spark/mnist_spark.py
--cluster_size ${SPARK_WORKER_INSTANCES}
--images examples/mnist/csv/train/images
--labels examples/mnist/csv/train/labels
--format csv
--mode train
--model mnist_model
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/06 18:02:05 INFO SparkContext: Running Spark version 1.6.0
17/11/06 18:02:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/06 18:02:05 WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.

Please instead use:

./spark-submit with --num-executors to specify the number of executors
Or set SPARK_EXECUTOR_INSTANCES
spark.executor.instances to configure the number of instances in the spark config.

17/11/06 18:02:05 INFO SecurityManager: Changing view acls to: root
17/11/06 18:02:05 INFO SecurityManager: Changing modify acls to: root
17/11/06 18:02:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
17/11/06 18:02:06 INFO Utils: Successfully started service 'sparkDriver' on port 42275.
17/11/06 18:02:06 INFO Slf4jLogger: Slf4jLogger started
17/11/06 18:02:06 INFO Remoting: Starting remoting
17/11/06 18:02:06 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@<IP_ADDRESS>:46798]
17/11/06 18:02:06 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 46798.
17/11/06 18:02:06 INFO SparkEnv: Registering MapOutputTracker
17/11/06 18:02:06 INFO SparkEnv: Registering BlockManagerMaster
17/11/06 18:02:06 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-de0db70a-8868-4abe-9bc9-e35b2ec3c9e6
17/11/06 18:02:06 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/11/06 18:02:06 INFO SparkEnv: Registering OutputCommitCoordinator
17/11/06 18:02:06 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/11/06 18:02:06 INFO SparkUI: Started SparkUI at http://<IP_ADDRESS>:4040
17/11/06 18:02:06 INFO HttpFileServer: HTTP File server directory is /tmp/spark-50b5e842-f624-475e-9038-eb4aa24955aa/httpd-7060d3e0-c97f-4ded-ba48-4213fa57eae7
17/11/06 18:02:06 INFO HttpServer: Starting HTTP Server
17/11/06 18:02:06 INFO Utils: Successfully started service 'HTTP file server' on port 44072.
17/11/06 18:02:06 INFO Utils: Copying /root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_spark.py to /tmp/spark-50b5e842-f624-475e-9038-eb4aa24955aa/userFiles-7c46538b-f166-4930-b265-3fb79e0bf334/mnist_spark.py
17/11/06 18:02:06 INFO SparkContext: Added file file:/root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_spark.py at http://<IP_ADDRESS>:44072/files/mnist_spark.py with timestamp<PHONE_NUMBER>541
17/11/06 18:02:06 INFO Utils: Copying /root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_dist.py to /tmp/spark-50b5e842-f624-475e-9038-eb4aa24955aa/userFiles-7c46538b-f166-4930-b265-3fb79e0bf334/mnist_dist.py
17/11/06 18:02:06 INFO SparkContext: Added file file:/root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_dist.py at http://<IP_ADDRESS>:44072/files/mnist_dist.py with timestamp<PHONE_NUMBER>545
17/11/06 18:02:06 INFO AppClient$ClientEndpoint: Connecting to master spark://I77700:7077...
17/11/06 18:02:06 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20171106180206-0000
17/11/06 18:02:06 INFO AppClient$ClientEndpoint: Executor added: app-20171106180206-0000/0 on worker-20171106180017-<IP_ADDRESS>-46452 (<IP_ADDRESS>:46452) with 1 cores
17/11/06 18:02:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20171106180206-0000/0 on hostPort <IP_ADDRESS>:46452 with 1 cores, 1024.0 MB RAM
17/11/06 18:02:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38011.
17/11/06 18:02:06 INFO NettyBlockTransferService: Server created on 38011
17/11/06 18:02:06 INFO BlockManagerMaster: Trying to register BlockManager
17/11/06 18:02:06 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:38011 with 511.1 MB RAM, BlockManagerId(driver, <IP_ADDRESS>, 38011)
17/11/06 18:02:06 INFO BlockManagerMaster: Registered BlockManager
17/11/06 18:02:06 INFO AppClient$ClientEndpoint: Executor updated: app-20171106180206-0000/0 is now RUNNING
17/11/06 18:02:06 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
args: Namespace(batch_size=100, cluster_size=1, epochs=1, format='csv', images='examples/mnist/csv/train/images', labels='examples/mnist/csv/train/labels', mode='train', model='mnist_model', output='predictions', rdma=False, readers=1, steps=1000, tensorboard=False)
2017-11-06T18:02:06.877654 ===== Start
17/11/06 18:02:07 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 208.5 KB, free 208.5 KB)
17/11/06 18:02:07 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.3 KB, free 227.8 KB)
17/11/06 18:02:07 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:38011 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:02:07 INFO SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:-2
17/11/06 18:02:07 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 208.5 KB, free 436.3 KB)
17/11/06 18:02:07 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 19.3 KB, free 455.6 KB)
17/11/06 18:02:07 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:38011 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:02:07 INFO SparkContext: Created broadcast 1 from textFile at NativeMethodAccessorImpl.java:-2
zipping images and labels
17/11/06 18:02:07 INFO FileInputFormat: Total input paths to process : 10
17/11/06 18:02:07 INFO FileInputFormat: Total input paths to process : 10
2017-11-06 18:02:07,275 INFO (MainThread-15426) Reserving TFSparkNodes
2017-11-06 18:02:07,276 INFO (MainThread-15426) listening for reservations at ('<IP_ADDRESS>', 36391)
2017-11-06 18:02:07,277 INFO (MainThread-15426) Starting TensorFlow on executors
2017-11-06 18:02:07,282 INFO (MainThread-15426) Waiting for TFSparkNodes to start
2017-11-06 18:02:07,282 INFO (MainThread-15426) waiting for 1 reservations
17/11/06 18:02:07 INFO SparkContext: Starting job: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253
17/11/06 18:02:07 INFO DAGScheduler: Got job 0 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253) with 1 output partitions
17/11/06 18:02:07 INFO DAGScheduler: Final stage: ResultStage 0 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253)
17/11/06 18:02:07 INFO DAGScheduler: Parents of final stage: List()
17/11/06 18:02:07 INFO DAGScheduler: Missing parents: List()
17/11/06 18:02:07 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[8] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253), which has no missing parents
17/11/06 18:02:07 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 10.5 KB, free 466.1 KB)
17/11/06 18:02:07 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.3 KB, free 473.4 KB)
17/11/06 18:02:07 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:38011 (size: 7.3 KB, free: 511.1 MB)
17/11/06 18:02:07 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
17/11/06 18:02:07 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (PythonRDD[8] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253)
17/11/06 18:02:07 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/11/06 18:02:07 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (I77700:46614) with ID 0
17/11/06 18:02:07 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, I77700, partition 0,PROCESS_LOCAL, 2196 bytes)
17/11/06 18:02:07 INFO BlockManagerMasterEndpoint: Registering block manager I77700:33973 with 511.1 MB RAM, BlockManagerId(0, I77700, 33973)
17/11/06 18:02:08 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on I77700:33973 (size: 7.3 KB, free: 511.1 MB)
2017-11-06 18:02:08,283 INFO (MainThread-15426) waiting for 1 reservations
2017-11-06 18:02:09,285 INFO (MainThread-15426) all reservations completed
2017-11-06 18:02:09,285 INFO (MainThread-15426) All TFSparkNodes started
2017-11-06 18:02:09,285 INFO (MainThread-15426) {'addr': '/tmp/pymp-9YnA8D/listener-Ro_DI8', 'task_index': 0, 'port': 35573, 'authkey': "\xd0\x9f\x12*%\F\xe1\xb3\x93\xa7'\xa4r\x81x", 'worker_num': 0, 'host': '<IP_ADDRESS>', 'ppid': 15640, 'job_name': 'worker', 'tb_pid': 0, 'tb_port': 0}
2017-11-06 18:02:09,286 INFO (MainThread-15426) Feeding training data
17/11/06 18:02:09 INFO SparkContext: Starting job: collect at PythonRDD.scala:405
17/11/06 18:02:09 INFO DAGScheduler: Got job 1 (collect at PythonRDD.scala:405) with 10 output partitions
17/11/06 18:02:09 INFO DAGScheduler: Final stage: ResultStage 1 (collect at PythonRDD.scala:405)
17/11/06 18:02:09 INFO DAGScheduler: Parents of final stage: List()
17/11/06 18:02:09 INFO DAGScheduler: Missing parents: List()
17/11/06 18:02:09 INFO DAGScheduler: Submitting ResultStage 1 (PythonRDD[10] at RDD at PythonRDD.scala:43), which has no missing parents
17/11/06 18:02:09 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 12.1 KB, free 485.5 KB)
17/11/06 18:02:09 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 6.4 KB, free 491.9 KB)
17/11/06 18:02:09 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on <IP_ADDRESS>:38011 (size: 6.4 KB, free: 511.1 MB)
17/11/06 18:02:09 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006
17/11/06 18:02:09 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 1 (PythonRDD[10] at RDD at PythonRDD.scala:43)
17/11/06 18:02:09 INFO TaskSchedulerImpl: Adding task set 1.0 with 10 tasks
17/11/06 18:02:09 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, I77700, partition 0,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:09 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1610 ms on I77700 (1/1)
17/11/06 18:02:09 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/11/06 18:02:09 INFO DAGScheduler: ResultStage 0 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253) finished in 2.251 s
17/11/06 18:02:09 INFO DAGScheduler: Job 0 finished: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253, took 2.278614 s
17/11/06 18:02:09 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on I77700:33973 (size: 6.4 KB, free: 511.1 MB)
17/11/06 18:02:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on I77700:33973 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:02:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on I77700:33973 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:02:14 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, I77700, partition 1,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:14 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 4459 ms on I77700 (1/10)
17/11/06 18:02:15 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, I77700, partition 2,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:15 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 1805 ms on I77700 (2/10)
17/11/06 18:02:17 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4, I77700, partition 3,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:17 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 1826 ms on I77700 (3/10)
17/11/06 18:02:19 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 5, I77700, partition 4,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:19 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 1798 ms on I77700 (4/10)
17/11/06 18:02:21 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 6, I77700, partition 5,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:21 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 5) in 1807 ms on I77700 (5/10)
17/11/06 18:02:23 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 7, I77700, partition 6,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:23 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 6) in 1805 ms on I77700 (6/10)
17/11/06 18:02:24 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 8, I77700, partition 7,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:24 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 7) in 1787 ms on I77700 (7/10)
17/11/06 18:02:26 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 9, I77700, partition 8,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:26 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 8) in 1827 ms on I77700 (8/10)
17/11/06 18:02:28 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 10, I77700, partition 9,PROCESS_LOCAL, 2785 bytes)
17/11/06 18:02:28 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 9) in 1788 ms on I77700 (9/10)
17/11/06 18:02:30 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 10) in 1655 ms on I77700 (10/10)
17/11/06 18:02:30 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
17/11/06 18:02:30 INFO DAGScheduler: ResultStage 1 (collect at PythonRDD.scala:405) finished in 20.788 s
17/11/06 18:02:30 INFO DAGScheduler: Job 1 finished: collect at PythonRDD.scala:405, took 20.798412 s
2017-11-06 18:02:30,116 INFO (MainThread-15426) Stopping TensorFlow nodes
17/11/06 18:02:30 INFO SparkContext: Starting job: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163
17/11/06 18:02:30 INFO DAGScheduler: Got job 2 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163) with 1 output partitions
17/11/06 18:02:30 INFO DAGScheduler: Final stage: ResultStage 2 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163)
17/11/06 18:02:30 INFO DAGScheduler: Parents of final stage: List()
17/11/06 18:02:30 INFO DAGScheduler: Missing parents: List()
17/11/06 18:02:30 INFO DAGScheduler: Submitting ResultStage 2 (PythonRDD[12] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163), which has no missing parents
17/11/06 18:02:30 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 6.0 KB, free 497.9 KB)
17/11/06 18:02:30 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.0 KB, free 501.9 KB)
17/11/06 18:02:30 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on <IP_ADDRESS>:38011 (size: 4.0 KB, free: 511.1 MB)
17/11/06 18:02:30 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/11/06 18:02:30 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (PythonRDD[12] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163)
17/11/06 18:02:30 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
17/11/06 18:02:30 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 11, I77700, partition 0,PROCESS_LOCAL, 2196 bytes)
17/11/06 18:02:30 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on I77700:33973 (size: 4.0 KB, free: 511.1 MB)
17/11/06 18:02:30 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 11) in 64 ms on I77700 (1/1)
17/11/06 18:02:30 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
17/11/06 18:02:30 INFO DAGScheduler: ResultStage 2 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163) finished in 0.064 s
17/11/06 18:02:30 INFO DAGScheduler: Job 2 finished: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163, took 0.068972 s
2017-11-06 18:02:30,193 INFO (MainThread-15426) Shutting down cluster
2017-11-06T18:02:35.204530 ===== Stop
17/11/06 18:02:35 INFO SparkContext: Invoking stop() from shutdown hook
17/11/06 18:02:35 INFO SparkUI: Stopped Spark web UI at http://<IP_ADDRESS>:4040
17/11/06 18:02:35 INFO SparkDeploySchedulerBackend: Shutting down all executors
17/11/06 18:02:35 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
17/11/06 18:02:35 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/11/06 18:02:35 INFO MemoryStore: MemoryStore cleared
17/11/06 18:02:35 INFO BlockManager: BlockManager stopped
17/11/06 18:02:35 INFO BlockManagerMaster: BlockManagerMaster stopped
17/11/06 18:02:35 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/11/06 18:02:35 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/11/06 18:02:35 INFO SparkContext: Successfully stopped SparkContext
17/11/06 18:02:35 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/11/06 18:02:35 INFO ShutdownHookManager: Shutdown hook called
17/11/06 18:02:35 INFO ShutdownHookManager: Deleting directory /tmp/spark-50b5e842-f624-475e-9038-eb4aa24955aa/httpd-7060d3e0-c97f-4ded-ba48-4213fa57eae7
17/11/06 18:02:35 INFO ShutdownHookManager: Deleting directory /tmp/spark-50b5e842-f624-475e-9038-eb4aa24955aa
17/11/06 18:02:35 INFO ShutdownHookManager: Deleting directory /tmp/spark-50b5e842-f624-475e-9038-eb4aa24955aa/pyspark-55481e14-5943-44ad-962e-a1ca4b6e7371
[root@I77700 TensorFlowOnSpark-master]# ls -l
total 320452
drwxrwxrwx.  6 root root      4096 Oct 26 10:38 docs
drwxrwxrwx.  6 root root        62 Oct 26 10:38 examples
-rwxrwxrwx.  1 root root      9210 Oct 26 10:38 LICENSE
drwxrwxrwx   2 root root       140 Nov  3 11:01 mnist
drwxr-xr-x   2 root root       276 Nov  6 18:02 mnist_model
-rwxrwxrwx.  1 root root      3215 Oct 26 10:38 README.md
drwxrwxrwx.  3 root root       121 Oct 26 10:38 scripts
-rwxrwxrwx.  1 root root        69 Oct 26 10:38 setup.cfg
-rwxrwxrwx.  1 root root       903 Oct 26 10:38 setup.py
drwxr-xr-x  14  500  500       234 Dec 21  2015 spark-1.6.0-bin-hadoop2.6
-rw-r--r--   1 root root 328110080 Dec 28  2015 spark-1.6.0-bin-hadoop2.6.tar
drwxrwxrwx.  2 root root         6 Oct 26 10:38 tensorflow
drwxrwxrwx.  2 root root      4096 Nov  4 13:32 tensorflowonspark
drwxrwxrwx.  2 root root       178 Oct 26 10:38 test
[root@I77700 TensorFlowOnSpark-master]# ls -l mnist_model
total 2000
-rw-r--r-- 1 root root    128 Nov  6 18:02 checkpoint
-rw-r--r-- 1 root root  82109 Nov  6 18:02 events.out.tfevents.1510020132.I77700
-rw-r--r-- 1 root root 116892 Nov  6 18:02 graph.pbtxt
-rw-r--r-- 1 root root 814164 Nov  6 18:02 model.ckpt-0.data-00000-of-00001
-rw-r--r-- 1 root root    372 Nov  6 18:02 model.ckpt-0.index
-rw-r--r-- 1 root root  43608 Nov  6 18:02 model.ckpt-0.meta
-rw-r--r-- 1 root root 814164 Nov  6 18:02 model.ckpt-345.data-00000-of-00001
-rw-r--r-- 1 root root    372 Nov  6 18:02 model.ckpt-345.index
-rw-r--r-- 1 root root  43608 Nov  6 18:02 model.ckpt-345.meta
[root@I77700 TensorFlowOnSpark-master]# ${SPARK_HOME}/bin/spark-submit \

--master ${MASTER}
--py-files ${TFoS_HOME}/examples/mnist/spark/mnist_dist.py
--conf spark.cores.max=${TOTAL_CORES}
--conf spark.task.cpus=${CORES_PER_WORKER}
--conf spark.executorEnv.JAVA_HOME="$JAVA_HOME"
${TFoS_HOME}/examples/mnist/spark/mnist_spark.py
--cluster_size ${SPARK_WORKER_INSTANCES}
--images examples/mnist/csv/test/images
--labels examples/mnist/csv/test/labels
--mode inference
--format csv
--model mnist_model
--output predictions
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/06 18:04:08 INFO SparkContext: Running Spark version 1.6.0
17/11/06 18:04:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/06 18:04:08 WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '1').
This is deprecated in Spark 1.0+.

Please instead use:

./spark-submit with --num-executors to specify the number of executors
Or set SPARK_EXECUTOR_INSTANCES
spark.executor.instances to configure the number of instances in the spark config.

17/11/06 18:04:08 INFO SecurityManager: Changing view acls to: root
17/11/06 18:04:08 INFO SecurityManager: Changing modify acls to: root
17/11/06 18:04:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
17/11/06 18:04:08 INFO Utils: Successfully started service 'sparkDriver' on port 46220.
17/11/06 18:04:08 INFO Slf4jLogger: Slf4jLogger started
17/11/06 18:04:08 INFO Remoting: Starting remoting
17/11/06 18:04:08 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@<IP_ADDRESS>:43995]
17/11/06 18:04:08 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 43995.
17/11/06 18:04:08 INFO SparkEnv: Registering MapOutputTracker
17/11/06 18:04:09 INFO SparkEnv: Registering BlockManagerMaster
17/11/06 18:04:09 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-a073ab53-0720-4379-9ba5-49fdbe20718f
17/11/06 18:04:09 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/11/06 18:04:09 INFO SparkEnv: Registering OutputCommitCoordinator
17/11/06 18:04:09 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/11/06 18:04:09 INFO SparkUI: Started SparkUI at http://<IP_ADDRESS>:4040
17/11/06 18:04:09 INFO HttpFileServer: HTTP File server directory is /tmp/spark-0097dc89-420e-438e-920d-9de397004ab7/httpd-abe647d6-8927-436c-a429-bc5a35325d4d
17/11/06 18:04:09 INFO HttpServer: Starting HTTP Server
17/11/06 18:04:09 INFO Utils: Successfully started service 'HTTP file server' on port 34087.
17/11/06 18:04:09 INFO Utils: Copying /root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_spark.py to /tmp/spark-0097dc89-420e-438e-920d-9de397004ab7/userFiles-0a0e9949-98e6-440d-925e-6cdf1401da8a/mnist_spark.py
17/11/06 18:04:09 INFO SparkContext: Added file file:/root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_spark.py at http://<IP_ADDRESS>:34087/files/mnist_spark.py with timestamp<PHONE_NUMBER>189
17/11/06 18:04:09 INFO Utils: Copying /root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_dist.py to /tmp/spark-0097dc89-420e-438e-920d-9de397004ab7/userFiles-0a0e9949-98e6-440d-925e-6cdf1401da8a/mnist_dist.py
17/11/06 18:04:09 INFO SparkContext: Added file file:/root/Downloads/TensorFlowOnSpark-master/examples/mnist/spark/mnist_dist.py at http://<IP_ADDRESS>:34087/files/mnist_dist.py with timestamp<PHONE_NUMBER>193
17/11/06 18:04:09 INFO AppClient$ClientEndpoint: Connecting to master spark://I77700:7077...
17/11/06 18:04:09 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20171106180409-0001
17/11/06 18:04:09 INFO AppClient$ClientEndpoint: Executor added: app-20171106180409-0001/0 on worker-20171106180017-<IP_ADDRESS>-46452 (<IP_ADDRESS>:46452) with 1 cores
17/11/06 18:04:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20171106180409-0001/0 on hostPort <IP_ADDRESS>:46452 with 1 cores, 1024.0 MB RAM
17/11/06 18:04:09 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33322.
17/11/06 18:04:09 INFO NettyBlockTransferService: Server created on 33322
17/11/06 18:04:09 INFO BlockManagerMaster: Trying to register BlockManager
17/11/06 18:04:09 INFO BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:33322 with 511.1 MB RAM, BlockManagerId(driver, <IP_ADDRESS>, 33322)
17/11/06 18:04:09 INFO BlockManagerMaster: Registered BlockManager
17/11/06 18:04:09 INFO AppClient$ClientEndpoint: Executor updated: app-20171106180409-0001/0 is now RUNNING
17/11/06 18:04:09 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
args: Namespace(batch_size=100, cluster_size=1, epochs=1, format='csv', images='examples/mnist/csv/test/images', labels='examples/mnist/csv/test/labels', mode='inference', model='mnist_model', output='predictions', rdma=False, readers=1, steps=1000, tensorboard=False)
2017-11-06T18:04:09.517530 ===== Start
17/11/06 18:04:09 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 208.5 KB, free 208.5 KB)
17/11/06 18:04:09 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.3 KB, free 227.8 KB)
17/11/06 18:04:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:33322 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:04:09 INFO SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:-2
17/11/06 18:04:09 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 208.5 KB, free 436.3 KB)
17/11/06 18:04:09 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 19.3 KB, free 455.6 KB)
17/11/06 18:04:09 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:33322 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:04:09 INFO SparkContext: Created broadcast 1 from textFile at NativeMethodAccessorImpl.java:-2
zipping images and labels
17/11/06 18:04:09 INFO FileInputFormat: Total input paths to process : 10
17/11/06 18:04:09 INFO FileInputFormat: Total input paths to process : 10
2017-11-06 18:04:09,893 INFO (MainThread-19741) Reserving TFSparkNodes
2017-11-06 18:04:09,894 INFO (MainThread-19741) listening for reservations at ('<IP_ADDRESS>', 45572)
2017-11-06 18:04:09,894 INFO (MainThread-19741) Starting TensorFlow on executors
2017-11-06 18:04:09,899 INFO (MainThread-19741) Waiting for TFSparkNodes to start
2017-11-06 18:04:09,899 INFO (MainThread-19741) waiting for 1 reservations
17/11/06 18:04:09 INFO SparkContext: Starting job: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253
17/11/06 18:04:09 INFO DAGScheduler: Got job 0 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253) with 1 output partitions
17/11/06 18:04:09 INFO DAGScheduler: Final stage: ResultStage 0 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253)
17/11/06 18:04:09 INFO DAGScheduler: Parents of final stage: List()
17/11/06 18:04:09 INFO DAGScheduler: Missing parents: List()
17/11/06 18:04:09 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[8] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253), which has no missing parents
17/11/06 18:04:09 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 10.5 KB, free 466.1 KB)
17/11/06 18:04:09 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 7.3 KB, free 473.4 KB)
17/11/06 18:04:09 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:33322 (size: 7.3 KB, free: 511.1 MB)
17/11/06 18:04:09 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
17/11/06 18:04:09 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (PythonRDD[8] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253)
17/11/06 18:04:09 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/11/06 18:04:10 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (I77700:39584) with ID 0
17/11/06 18:04:10 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, I77700, partition 0,PROCESS_LOCAL, 2196 bytes)
17/11/06 18:04:10 INFO BlockManagerMasterEndpoint: Registering block manager I77700:35219 with 511.1 MB RAM, BlockManagerId(0, I77700, 35219)
17/11/06 18:04:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on I77700:35219 (size: 7.3 KB, free: 511.1 MB)
2017-11-06 18:04:10,900 INFO (MainThread-19741) waiting for 1 reservations
2017-11-06 18:04:11,901 INFO (MainThread-19741) all reservations completed
2017-11-06 18:04:11,901 INFO (MainThread-19741) All TFSparkNodes started
2017-11-06 18:04:11,901 INFO (MainThread-19741) {'addr': '/tmp/pymp-p6z0Ga/listener-PPhBSZ', 'task_index': 0, 'port': 35183, 'authkey': '\xa3u\x8d:H\xa0@%\x8b\t0Bem\x8a:', 'worker_num': 0, 'host': '<IP_ADDRESS>', 'ppid': 19955, 'job_name': 'worker', 'tb_pid': 0, 'tb_port': 0}
2017-11-06 18:04:11,902 INFO (MainThread-19741) Feeding inference data
17/11/06 18:04:11 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/11/06 18:04:11 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/11/06 18:04:11 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/11/06 18:04:11 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/11/06 18:04:11 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/11/06 18:04:11 INFO SparkContext: Starting job: saveAsTextFile at NativeMethodAccessorImpl.java:-2
17/11/06 18:04:11 INFO DAGScheduler: Got job 1 (saveAsTextFile at NativeMethodAccessorImpl.java:-2) with 10 output partitions
17/11/06 18:04:11 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsTextFile at NativeMethodAccessorImpl.java:-2)
17/11/06 18:04:11 INFO DAGScheduler: Parents of final stage: List()
17/11/06 18:04:11 INFO DAGScheduler: Missing parents: List()
17/11/06 18:04:11 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[11] at saveAsTextFile at NativeMethodAccessorImpl.java:-2), which has no missing parents
17/11/06 18:04:11 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 72.4 KB, free 545.8 KB)
17/11/06 18:04:11 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 26.6 KB, free 572.4 KB)
17/11/06 18:04:11 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on <IP_ADDRESS>:33322 (size: 26.6 KB, free: 511.1 MB)
17/11/06 18:04:11 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006
17/11/06 18:04:11 INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 1 (MapPartitionsRDD[11] at saveAsTextFile at NativeMethodAccessorImpl.java:-2)
17/11/06 18:04:11 INFO TaskSchedulerImpl: Adding task set 1.0 with 10 tasks
17/11/06 18:04:12 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, I77700, partition 0,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:12 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1527 ms on I77700 (1/1)
17/11/06 18:04:12 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/11/06 18:04:12 INFO DAGScheduler: ResultStage 0 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253) finished in 2.160 s
17/11/06 18:04:12 INFO DAGScheduler: Job 0 finished: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:253, took 2.187191 s
17/11/06 18:04:12 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on I77700:35219 (size: 26.6 KB, free: 511.1 MB)
17/11/06 18:04:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on I77700:35219 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:04:12 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on I77700:35219 (size: 19.3 KB, free: 511.1 MB)
17/11/06 18:04:15 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, I77700, partition 1,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:15 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 3023 ms on I77700 (1/10)
17/11/06 18:04:15 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, I77700, partition 2,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:15 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 431 ms on I77700 (2/10)
17/11/06 18:04:15 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 4, I77700, partition 3,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:15 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 420 ms on I77700 (3/10)
17/11/06 18:04:16 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 5, I77700, partition 4,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:16 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 4) in 426 ms on I77700 (4/10)
17/11/06 18:04:16 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 6, I77700, partition 5,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:16 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 5) in 426 ms on I77700 (5/10)
17/11/06 18:04:17 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 7, I77700, partition 6,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:17 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 6) in 414 ms on I77700 (6/10)
17/11/06 18:04:17 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 8, I77700, partition 7,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:17 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 7) in 421 ms on I77700 (7/10)
17/11/06 18:04:18 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 9, I77700, partition 8,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:18 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 8) in 420 ms on I77700 (8/10)
17/11/06 18:04:18 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 10, I77700, partition 9,PROCESS_LOCAL, 2674 bytes)
17/11/06 18:04:18 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 9) in 431 ms on I77700 (9/10)
17/11/06 18:04:18 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 10) in 434 ms on I77700 (10/10)
17/11/06 18:04:18 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
17/11/06 18:04:18 INFO DAGScheduler: ResultStage 1 (saveAsTextFile at NativeMethodAccessorImpl.java:-2) finished in 6.941 s
17/11/06 18:04:18 INFO DAGScheduler: Job 1 finished: saveAsTextFile at NativeMethodAccessorImpl.java:-2, took 6.953460 s
2017-11-06 18:04:18,949 INFO (MainThread-19741) Stopping TensorFlow nodes
17/11/06 18:04:18 INFO SparkContext: Starting job: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163
17/11/06 18:04:18 INFO DAGScheduler: Got job 2 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163) with 1 output partitions
17/11/06 18:04:18 INFO DAGScheduler: Final stage: ResultStage 2 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163)
17/11/06 18:04:18 INFO DAGScheduler: Parents of final stage: List()
17/11/06 18:04:18 INFO DAGScheduler: Missing parents: List()
17/11/06 18:04:18 INFO DAGScheduler: Submitting ResultStage 2 (PythonRDD[13] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163), which has no missing parents
17/11/06 18:04:18 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 6.0 KB, free 578.3 KB)
17/11/06 18:04:18 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.0 KB, free 582.3 KB)
17/11/06 18:04:18 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on <IP_ADDRESS>:33322 (size: 4.0 KB, free: 511.1 MB)
17/11/06 18:04:18 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/11/06 18:04:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (PythonRDD[13] at foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163)
17/11/06 18:04:18 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
17/11/06 18:04:18 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 11, I77700, partition 0,PROCESS_LOCAL, 2196 bytes)
17/11/06 18:04:18 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on I77700:35219 (size: 4.0 KB, free: 511.1 MB)
17/11/06 18:04:19 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 11) in 58 ms on I77700 (1/1)
17/11/06 18:04:19 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
17/11/06 18:04:19 INFO DAGScheduler: ResultStage 2 (foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163) finished in 0.058 s
17/11/06 18:04:19 INFO DAGScheduler: Job 2 finished: foreachPartition at /usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py:163, took 0.063344 s
2017-11-06 18:04:19,024 INFO (MainThread-19741) Shutting down cluster
2017-11-06T18:04:24.034610 ===== Stop
17/11/06 18:04:24 INFO SparkContext: Invoking stop() from shutdown hook
17/11/06 18:04:24 INFO SparkUI: Stopped Spark web UI at http://<IP_ADDRESS>:4040
17/11/06 18:04:24 INFO SparkDeploySchedulerBackend: Shutting down all executors
17/11/06 18:04:24 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
17/11/06 18:04:24 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/11/06 18:04:24 INFO MemoryStore: MemoryStore cleared
17/11/06 18:04:24 INFO BlockManager: BlockManager stopped
17/11/06 18:04:24 INFO BlockManagerMaster: BlockManagerMaster stopped
17/11/06 18:04:24 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/11/06 18:04:24 INFO SparkContext: Successfully stopped SparkContext
17/11/06 18:04:24 INFO ShutdownHookManager: Shutdown hook called
17/11/06 18:04:24 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/11/06 18:04:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-0097dc89-420e-438e-920d-9de397004ab7/httpd-abe647d6-8927-436c-a429-bc5a35325d4d
17/11/06 18:04:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-0097dc89-420e-438e-920d-9de397004ab7/pyspark-1b041553-d2d7-4fa6-ac10-ac35657291a7
17/11/06 18:04:24 INFO ShutdownHookManager: Deleting directory /tmp/spark-0097dc89-420e-438e-920d-9de397004ab7
17/11/06 18:04:24 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
[root@I77700 TensorFlowOnSpark-master]# less predictions/part-00000
[root@I77700 TensorFlowOnSpark-master]# ll
total 320456
drwxrwxrwx.  6 root root      4096 Oct 26 10:38 docs
drwxrwxrwx.  6 root root        62 Oct 26 10:38 examples
-rwxrwxrwx.  1 root root      9210 Oct 26 10:38 LICENSE
drwxrwxrwx   2 root root       140 Nov  3 11:01 mnist
drwxr-xr-x   2 root root       321 Nov  6 18:04 mnist_model
drwxr-xr-x   2 root root      4096 Nov  6 18:04 predictions
-rwxrwxrwx.  1 root root      3215 Oct 26 10:38 README.md
drwxrwxrwx.  3 root root       121 Oct 26 10:38 scripts
-rwxrwxrwx.  1 root root        69 Oct 26 10:38 setup.cfg
-rwxrwxrwx.  1 root root       903 Oct 26 10:38 setup.py
drwxr-xr-x  14  500  500       234 Dec 21  2015 spark-1.6.0-bin-hadoop2.6
-rw-r--r--   1 root root 328110080 Dec 28  2015 spark-1.6.0-bin-hadoop2.6.tar
drwxrwxrwx.  2 root root         6 Oct 26 10:38 tensorflow
drwxrwxrwx.  2 root root      4096 Nov  4 13:32 tensorflowonspark
drwxrwxrwx.  2 root root       178 Oct 26 10:38 test
[root@I77700 TensorFlowOnSpark-master]#


[Rendering] "associatedInputs": "none" doesn't work for Action.Execute in Microsoft Teams for iOS

Target Platforms
iOS
SDK Version
1.4
Application Name
Microsoft Teams
Problem Description
Parameter "associatedInputs": "none" doesn't work for "Action.Execute" type of button in Microsoft Teams for iOS.
Works perfectly well on Desktop and Android.
Works for "Action.Submit" on all devices.
1.5 SDK - same.
Screenshots

Card JSON
{
    "type": "AdaptiveCard",
    "body": [
        {
            "type": "ColumnSet",
            "columns": [
                {
                    "type": "Column",
                    "width": "stretch",
                    "items": [
                        {
                            "type": "TextBlock",
                            "wrap": true,
                            "text": "Leave Request",
                            "size": "Large",
                            "weight": "Bolder"
                        }
                    ],
                    "verticalContentAlignment": "Center"
                }
            ],
            "horizontalAlignment": "Left"
        },
        {
            "type": "Input.ChoiceSet",
            "choices": [
                {
                    "title": "Annual Leave",
                    "value": "BasicHoliday"
                },
                {
                    "title": "Unpaid Leave",
                    "value": "UnpaidHoliday"
                }
            ],
            "placeholder": "Choose",
            "label": "Type of Leave",
            "isRequired": true,
            "errorMessage": "Select type of leave",
            "id": "typeLeave",
            "spacing": "Large",
            "value": "${typeOfLeave}",
            "separator": true
        },
        {
            "type": "Input.Date",
            "id": "startD",
            "label": "Start date",
            "isRequired": true,
            "errorMessage": "Select a date",
            "value": "${startDateStr}"
        },
        {
            "type": "Input.Date",
            "id": "endD",
            "label": "End date",
            "isRequired": true,
            "errorMessage": "Select a date",
            "value": "${endDateStr}"
        }
    ],
    "actions": [
        {
            "type": "Action.Execute",
            "title": "Submit",
            "verb": "hdcreateticket",
            "data": {
                "action": "create",
                "basicHoliday": "${basicHoliday}",
                "unpaidHoliday": "${unpaidHoliday}",
                "userEmail": "${userEmail}"
            }
        },
        {
            "type": "Action.Execute",
            "title": "Cancel",
            "verb": "createticket",
            "associatedInputs": "none",
            "data": {
                "action": "cancel",
                "basicHoliday": "${basicHoliday}",
                "unpaidHoliday": "${unpaidHoliday}",
                "userEmail": "${userEmail}"
            }
        }
    ],
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
    "version": "1.4"
}

Sample Code Language
No response
Sample Code
No response

This issue has been copied over to Microsoft Teams repo here: https://github.com/MicrosoftDocs/msteams-docs/issues/8956


Serving pipeline: write PagedCrossReferences to the resulting serving table [phab:D641]

Created by schroederc at 2015-12-10 20:48:13:
The PagedCrossReferences structure is similiar to a PagedEdgeSet with a
top-level message used as the root for the CrossReferences/Edges lookup
and zero-or-mode Pages stored separately to reduce the size of a single
lookup.  The pipeline constructs the new data using the same SetPager as
the EdgeSetBuilder.
This change only contains the logic to write cross-references data to
the serving table.  A separate change will have the logic to read/serve
the data.

zarko accepted this revision at 2015-12-10 22:40:10

schroederc closed via commit 42ffe891899f6760f58efbe8e79332aad7e02082 at 2015-12-10 23:28:36


[BUG] pydantic 2.5 not support?

Describe the bug
import fastapi_restfule failed
If you can generate a relevant traceback, please include it.
(.test_env) caoyizhendeMacBook-Air➜  test  ᐅ   cd /Users/caoyizhen/Documents/python代码/test ; /usr/bin/env /Users/caoyizhen/Documents/python代码/te
st/.test_env/bin/python /Users/caoyizhen/.vscode/extensions/ms-python.python-2023.22.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher
50436 -- /Users/caoyizhen/Documents/python代码/test/test.py
Traceback (most recent call last):
File "/Users/caoyizhen/Documents/python代码/test/test.py", line 109, in
from fastapi_restful import Api, Resource
File "/Users/caoyizhen/Documents/python代码/test/.test_env/lib/python3.9/site-packages/fastapi_restful/init.py", line 1, in
from .cbv_base import Api, Resource, set_responses, take_init_parameters
File "/Users/caoyizhen/Documents/python代码/test/.test_env/lib/python3.9/site-packages/fastapi_restful/cbv_base.py", line 5, in
from .cbv import INCLUDE_INIT_PARAMS_KEY, RETURN_TYPES_FUNC_KEY, _cbv
File "/Users/caoyizhen/Documents/python代码/test/.test_env/lib/python3.9/site-packages/fastapi_restful/cbv.py", line 11, in
from typing_inspect import is_classvar
ModuleNotFoundError: No module named 'typing_inspect'
To Reproduce
Steps to reproduce the behavior:

create dir  code : mkdir test
code: cd test   ->  python -m venv .env
source .env/bin/activate
pip install fastapi pydantic fastapi-restful
create a python file
import fastapi_restful

Expected behavior
A clear and concise description of what you expected to happen.
code can be run
Screenshots
If applicable, add screenshots to help explain your problem.

Environment:

OS: macos
FastAPI RESTful, FastAPI, and Pydantic versions [e.g. 0.3.0], get them with:
fastapi            0.108.0
fastapi-restful    0.5.0
pydantic           2.5.3
pydantic_core      2.14.6

from fastapi import FastAPI
from fastapi.requests import Request
from fastapi_restful import Api, Resource
from pydantic import BaseModel
import uvicorn

app = FastAPI()
api = Api(app)
class Model(BaseModel):
    name : str
    age : int

class Index(Resource):
    async def post(self, request:Request):
        data = await request.json()
        data = Model(**data)
        return data

api.add_resource(Index, "/index")

if __name__ == "__main__":
    uvicorn.run(app)

Python version, get it with:

Python 3.9.6

Additional context
Add any other context about the problem here.

when I  pip install typing-inspect then error message TypeError: can't set attributes of built-in/extension type 'type'

Sorry I forgot to instantiate my Resource


multiple excludes doesn't seem to work

hi
2 things really, using .Net Core 2.1, Dev Ops and MSBuild
We have a test project that for some reason seems to be included in the code coverage report whereas other test projects seems to be fine, not sure why this.
Trying to exclude this project along with a class library one that is used to generate test data
we apply the following parameter
/p:Exclude="[.TestFramework],[]XXX.XXXX.Services.Ontology.Common.Tests.*"
this results in none of the tests being run and also no coverage produced, if i choose one or the other statement then they get excluded correctly.
any ideas ?

Some questions... which version of coverlet are you using?
Can you try to enable logging to understand what's happening? https://github.com/tonerdo/coverlet/blob/master/Documentation/Troubleshooting.md

version is 2.6.3, will enable logging tomorrow once i get into work

Seems to be related to ms build issue. Provided workaround works for me: /p:Exclude="[Org.Assembly1.*]*%2c[Org.Assembly2.*]*"

Thank's @aleksandrshvets for tip @markwest51 do you have news?

apologies, did not get any notification.  Yes this resolves the issue, have been told by a fellow developer at work today that it does exclude multiple libraries using %2c.  Some reason the other issue of having a test project appear in coverage seems to have gone during further changes in the solution, not sure why but it looks good now

Yes this resolves the issue,

Glad to hear, thanks @aleksandrshvets for solving!


[CI] base image multiple arch support

tested with arm64 https://github.com/sustainable-computing-io/kepler/actions/runs/7422356767/job/20197478578?pr=1169

@rootfs , it will take lots of time to build base image on buildx... any suggestion?

@rootfs , could you please help with a s390 image belowing?
#2 [linux/s390 internal] load metadata for registry.access.redhat.com/ubi9/go-toolset:1.20
#2 ERROR: no match for platform in manifest: not found

cc: @jiangphcn , @jichenjc

https://catalog.redhat.com/software/containers/ubi9/618326f8c0d15aff4912fe0b?architecture=s390x&image=6571494d263ad9efd38089c4&container-tabs=gti
@rootfs , could you please help check the reason for s390x image fail with docker buildx? https://github.com/sustainable-computing-io/kepler/actions/runs/7423083117/job/20199711813?pr=1169

@vimalk78 can you take a look?

@rootfs , @vimalk78
I just submit changes according to pr review suggestion. It usually takes 1 hour for the job to be completed. I will take some personal issue and may absent from github in days(maybe...) hence
as following items for this PR.

if there is no further review comments, I will set CI hooks back and make the PR ready for merge.
Once merged, we need keep eye on it to see if it works.
If base image works well, another PR for multiple arch should be submitted and merged.


clear-pr-merge-commit-message should preserve which issues a PR closes when it is not merged into the default branch

I'm looking for a good way to keep a linear commit history, and work on new release branches while being able to preserve GitHub's auto-close feature for issues when a PR gets merged into master. My plan is to merge PRs (I'm squash-merging) into a develop branch and when I deem it to be ready, locally rebase it on top of master, then fast-forward master. Unfortunately the rebase step (or if I were to squash-merge the branch), GitHub forgets which issues should be closed.
If the commit bodies would contain the information of which issues get closed, for example
fixes: #1234
fixes: #4567
...

the commits would still cause Github to close the referenced issues, no matter how the commits finally end up in master.
To summarize, I'm asking for a way to automatically put the linked issues of a PR in the final commit message when the PR is not merged into the default branch. Currently, clear-pr-merge-commit-message clears everything but Co-authored-by: <...>

To summarize, I'm asking for a way to automatically put the linked issues of a PR in the final commit message when the PR is not merged into the default branch.

So if I'm not mistaken, this is a not a bug, but a new feature request? Because afaik GitHub doesn't natively include references to the closed issues in the commit description.

I'd consider it a feature request, yes.

This just needs a regex to detect and preserve these fixes/resolves/fix/etc keywords. I consider it a bug as it's breaking native GH behavior

I consider it a bug as it's breaking native GH behavior

But afaict, GitHub doesn't automatically add this info to the "extended description" box in the first place.

I consider it a bug as it's breaking native GH behavior

But afaict, GitHub doesn't automatically add this info to the "extended description" box in the first place.

It's true that if you link the issue via UI or "fixes" keyword in a comment, they won't add it to the description.
But they do when fixes #n appears in the commit description itself. This situation is relatively niche, but it's also a regex away. I suppose the regex would be kinda complex however, like:
(fixe?s?|resolves?|closes?) (issueRegex|urlRegex)

https://www.npmjs.com/package/issue-regex
https://www.npmjs.com/package/url-regex

Maybe https://www.npmjs.com/package/regex-join could also help

Don't understand why we cant just do on the first post and extract the issue number
	select.all('[aria-label="This pull request closes issue*"]')

It wouldn't preserve the same wording but that's totally fine for me. 💙
We need to verify if this is still happening though, I'd kinda be surprised that GitHub doesn't close the issues regardless of the branch, especially with the new "linked issues" in the sidebar. Do they have the same behavior?

The only way to check is by trying it out 😀
If I remember correctly they only close if its the default branch by design.

https://github.com/refined-github/sandbox/pull/52
It does not close it

Great, thanks!


Player Does't Use Latest Stream

(Correct me if I'm wrong) I think the current player implementation takes the first available video for an Eth address and makes that the stream for the channel.
Because of the way our p2p network works, there is a problem with that - a new video stream will not be on the gateway node (exit node hooked up to the CDN), and therefore initially appear like it's not available.  So if someone stops an old stream and immediately starts a new one, the old stream will appear to be still available in the CDN, and the new stream will appear to be unavailable.
One possible work-around is to always try and show the video with the highest JobID.
p.s - The stream usually becomes available after 5-15 seconds.

It should show the video with the highest jobId if it's on the CDN. The current player doesn't know anything about the p2p network and has no way of loading videos if they aren't on the CDN.


Fix code example for SSL Errors

Hello!:)
This is probably my smallest pull request ever. 😎
I just noticed that the code example for DangerousAcceptAnyServerCertificateValidator is broken. I hope it is fine that I didn't bother opening an issue first, this PR message is already about 2 orders of magnitude longer than the actual change I had to make to the rst. 😉
Proposed Changes

Fix the code example, show that DangerousAcceptAnyServerCertificateValidator has to be set to true to disable certification validation, rather than false.

Good catch!!!