Installing Tensorflow with GPU support on os x 10.11

After configuring and running
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
I am getting this error during the build:
ERROR: /pathtotensorflow/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1: Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. Traceback (most recent call last):   File "/private/var/tmp/_bazel_xxx/2ba3b3e08313f86d16eb1de7b54bf064/tensorflow/bazel-out/host/bin/tensorflow/contrib/session_bundle/example/export_half_plus_two.runfiles/tensorflow/contrib/session_bundle/example/export_half_plus_two.py", line 33, in <module>     from tensorflow.contrib.session_bundle import exporter ImportError: No module named session_bundle Target //tensorflow/tools/pip_package:build_pip_package failed to build
Environment info
Operating System: os x el Capitan
Installed version of CUDA and cuDNN:
-rwxr-xr-x  1 root  wheel  8280 Apr 12 23:02 /usr/local/cuda/lib/libcuda.dylib lrwxr-xr-x  1 root  wheel    45 Apr 12 23:03 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a lrwxr-xr-x  1 root  wheel    50 Apr 12 23:03 /usr/local/cuda/lib/libcudart.7.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib lrwxr-xr-x  1 root  wheel    46 Apr 12 23:03 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib lrwxr-xr-x  1 root  wheel    49 Apr 12 23:03 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a lrwxr-xr-x  1 root  admin    47 Jun 16 11:25 /usr/local/cuda/lib/libcudnn.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.5.dylib lrwxr-xr-x  1 root  admin    45 Jun 16 11:25 /usr/local/cuda/lib/libcudnn.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.dylib lrwxr-xr-x  1 root  admin    48 Jun 16 11:25 /usr/local/cuda/lib/libcudnn_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn_static.a

Hey guys
I actually pass the bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package process.
But there is no wheel file in the bazel-bin/tensorflow/tools/pip_package/ and
the bazel-bin/tensorflow/tools/pip_package/build_pip_package is not a directory.
Anyone has any idear?

You have to run bazel-bin/tensorflow/tools/pip_package/build_pip_package.
See the installation instructions.
On Tue, Jun 21, 2016 at 06:28 Velkan.Xu<EMAIL_ADDRESS>wrote:

Hey guys
I actually pass the bazel build -c opt --config=cuda
//tensorflow/tools/pip_package:build_pip_package process.
But there is no wheel file in the bazel-bin/tensorflow/tools/pip_package/
and
the bazel-bin/tensorflow/tools/pip_package/build_pip_package is not a
directory.
Anyone has any idear?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2940#issuecomment-227439086,
or mute the thread
https://github.com/notifications/unsubscribe/AAjO_Ym-8fCQzn1kP95IqZ1S5I7OZMl2ks5qN-b0gaJpZM4I4z4H
.

Hi Martin:
Thanks to reply this, really.
I run 'bazel-bin/tensorflow/tools/pip_package/build_pip_package' but got a
error 'error: invalid command 'bdist_wheel''
I spent a day googled it and some said I need to install the wheel and
setuptools package which I already did. And this error still there.
Some said maybe there was a conflict between the old version of setuptools
and pip, but setuptools and pip on my macbook were both updated. And the error
still there.
There is a closed issue: https://github.com/tensorflow/tensorflow/issues/348
This guy andyyuan78 https://github.com/andyyuan78 said that he got this
error before and solved it by run 'pip2 install wheel'.
I tried both 'pip install' and 'pip3 install' but it didn't work.
I'm using Python3.5 and this is the only Python I have on my macbook.
Any ideas?
Martin Wicke<EMAIL_ADDRESS>上午11:32写道：

You have to run bazel-bin/tensorflow/tools/pip_package/build_pip_package.
See the installation instructions.
On Tue, Jun 21, 2016 at 06:28 Velkan.Xu<EMAIL_ADDRESS>wrote:

Hey guys
I actually pass the bazel build -c opt --config=cuda
//tensorflow/tools/pip_package:build_pip_package process.
But there is no wheel file in the bazel-bin/tensorflow/tools/pip_package/
and
the bazel-bin/tensorflow/tools/pip_package/build_pip_package is not a
directory.
Anyone has any idear?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<
https://github.com/tensorflow/tensorflow/issues/2940#issuecomment-227439086
,
or mute the thread
<
https://github.com/notifications/unsubscribe/AAjO_Ym-8fCQzn1kP95IqZ1S5I7OZMl2ks5qN-b0gaJpZM4I4z4H
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2940#issuecomment-227940782,
or mute the thread
https://github.com/notifications/unsubscribe/ADmW7B6IUAM_ftseeymPcaSblzZs3PIMks5qOf5IgaJpZM4I4z4H
.

I ran into a similar issue with Python 2.7.11 on OS X 10.11.5 + MBP.
$ ./configure
Please specify the location of python. [Default is /usr/local/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] N
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc nvcc should use as the host compiler. [Default is /usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 7.5
Please specify the location where CUDA 7.5 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.

INFO: From Linking tensorflow/python/_pywrap_tensorflow.so [for host]:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
INFO: From Linking tensorflow/python/_pywrap_tensorflow.so:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
ERROR: /Users/peter_wang/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1: Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 245.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.7.5.dylib locally
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1826.337s, Critical Path: 1821.08s
To verify CUDA install:
$ ./bin/x86_64/darwin/release/deviceQuery
./bin/x86_64/darwin/release/deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GT 650M"
CUDA Driver Version / Runtime Version          7.5 / 7.5
CUDA Capability Major/Minor version number:    3.0
Total amount of global memory:                 512 MBytes (536543232 bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP:     384 CUDA Cores
GPU Max Clock rate:                            405 MHz (0.41 GHz)
Memory Clock rate:                             2000 Mhz
Memory Bus Width:                              128-bit
L2 Cache Size:                                 262144 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z):<PHONE_NUMBER>, 65535, 65535)
Maximum memory pitch:<PHONE_NUMBER> bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
Run time limit on kernels:                     Yes
Integrated GPU sharing Host Memory:            No
Support host page-locked memory mapping:       Yes
Alignment requirement for Surfaces:            Yes
Device has ECC support:                        Disabled
Device supports Unified Addressing (UVA):      Yes
Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = GeForce GT 650M
Result = PASS

Hi peter:
I actually pass the build process.
Maybe you want set a cuda capability during the tensorflow configuration?
When it like:
'Please specify a list of comma-separated Cuda compute capabilities you
want to build with.'
Enter '3.0' for the gt 650m.
And your error log said:
'Use --verbose_failures to see the command lines of failed build steps.'
Maybe you can get some details by doing that?
And make sure your source code is updated. I got a wired error once by
using a old version of the source and solved it by clone from the master.
On Fri, Jun 24, 2016, 3:04 AM peterswang<EMAIL_ADDRESS>wrote:

I ran into a similar issue with Python 2.7.11 on OS X 10.11.5 + MBP.
$ ./configure
Please specify the location of python. [Default is /usr/local/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] N
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc nvcc should use as the host compiler. [Default is
/usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave
empty to use system default]: 7.5
Please specify the location where CUDA 7.5 toolkit is installed. Refer to
README.md for more details. [Default is /usr/local/cuda]:
Please specify the Cudnn version you want to use. [Leave empty to use
system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to
README.md for more details. [Default is /usr/local/cuda]:
Please specify a list of comma-separated Cuda compute capabilities you
want to build with.
You can find the compute capability of your device at:
https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly
increases your build time and binary size.

INFO: From Linking tensorflow/python/_pywrap_tensorflow.so [for host]:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
INFO: From Linking tensorflow/python/_pywrap_tensorflow.so:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
ERROR:
/Users/peter_wang/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1:
Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two
failed: bash failed: error executing command /bin/bash -c ... (remaining 1
argument(s) skipped):
com.google.devtools.build.lib.shell.BadExitStatusException: Process exited
with status 245.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA
library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA
library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA
library libcufft.7.5.dylib locally
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1826.337s, Critical Path: 1821.08s
To verify CUDA install:
$ ./bin/x86_64/darwin/release/deviceQuery
./bin/x86_64/darwin/release/deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GT 650M"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 3.0
Total amount of global memory: 512 MBytes (536543232 bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 405 MHz (0.41 GHz)
Memory Clock rate: 2000 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 262144 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536),
3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048
layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z):<PHONE_NUMBER>, 65535, 65535)
Maximum memory pitch:<PHONE_NUMBER> bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device
simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime
Version = 7.5, NumDevs = 1, Device0 = GeForce GT 650M
Result = PASS
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/2940#issuecomment-228151515,
or mute the thread
https://github.com/notifications/unsubscribe/ADmW7AoIow1wLqEj_Jk1rli5ifANu3Zlks5qOtjUgaJpZM4I4z4H
.

Thx for your suggestions, Velkan. I followed the instructions under the Build from Source section of the TF [Download and Setup] page. Compute capability was specified to be '3.0', just that the copy-paste missed that line.
The Github TF page shows last successful build for Mac OS to be 14 days ago. I just tried building again with a fresh clone from the repo and it crashed at the same place. Actually, bazel apparently invoked Python, which crashed, seemingly while trying to access the CUDA library, perhaps for libcufftw.7.5.dylib?
If it may be of any help to Martin, below is part of call stack from the crash report:
...
Crashed Thread:        0  Dispatch queue: com.apple.main-thread
Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
VM Regions Near 0:
-->
__TEXT                 000000010aec3000-000000010aec5000 [    8K] r-x/rwx SM=COW  /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib             	0x00007fff8e610152 strlen + 18
1   _pywrap_tensorflow.so         	0x000000010d3e19a2 perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 402
2   _pywrap_tensorflow.so         	0x000000010d3e2692 perftools::gputools::internal::DsoLoader::GetLibcudaDsoHandle(void**) + 258
3   _pywrap_tensorflow.so         	0x000000010d0d1f22 std::__1::__function::__func<tensorflow::Status ()(tensorflow::shape_inference::InferenceContext), std::__1::allocator<tensorflow::Status ()(tensorflow::shape_inference::InferenceContext)>, tensorflow::Status (tensorflow::shape_inference::InferenceContext*)>::operator()(tensorflow::shape_inference::InferenceContext*&&) + 18
4   _pywrap_tensorflow.so         	0x000000010d3e2f7f perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so         	0x000000010d3e34e0 perftools::gputools::internal::CachedDsoLoader::GetLibcudaDsoHandle() + 112
6   _pywrap_tensorflow.so         	0x000000010d464fe5 perftools::gputools::initialize_cuda_gpu_executor() + 37
7   dyld                          	0x00007fff60b5a10b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265
8   dyld                          	0x00007fff60b5a284 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                          	0x00007fff60b568bd ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305
10  dyld                          	0x00007fff60b56743 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 127
11  dyld                          	0x00007fff60b569b3 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                          	0x00007fff60b4beb0 dyld::runInitializers(ImageLoader*) + 89
13  dyld                          	0x00007fff60b53308 dlopen + 555
14  libdyld.dylib                 	0x00007fff8d7ae79c dlopen + 59
15  org.python.python             	0x000000010af807d9 _PyImport_GetDynLoadFunc + 309
...

Just another datapoint. I'm getting the same python crash using
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
After Python crashes for bazel build near the end, then
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
gives the error
error: invalid command 'bdist_wheel'
(Note: the directory /tmp/tensorflow_pkg does not exist. Probably because of the python crash)
Possible useful config information:
System:
  System Version:	OS X 10.10.5 (14F1808)
  Kernel Version:	Darwin 14.5.0
  Chipset Model:	NVIDIA GeForce GTX 760
  Type:	GPU
  Bus:	PCIe
  PCIe Lane Width:	x16
  VRAM (Total):	4096 MB
  Vendor:	NVIDIA (0x10de)
  Device ID:	0x1187
  Revision ID:	0x00a2
  ROM Revision:	preset 1.0.0

> python --version
Python 2.7.11

> /Developer/NVIDIA/CUDA-7.5/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Thu_Sep_24_00:26:39_CDT_2015
Cuda compilation tools, release 7.5, V7.5.19

>pip --version
pip 8.1.2 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)

> bazel version
Build label: 0.2.3-homebrew
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue May 17 15:07:33 2016<PHONE_NUMBER>)
Build timestamp:<PHONE_NUMBER>
Build timestamp as int:<PHONE_NUMBER>

Crash stack
Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000

VM Regions Near 0:
-->
    __TEXT<PHONE_NUMBER>000000-0000000100001000 [    4K] r-x/rwx SM=COW  /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib             	0x00007fff9666cf92 strlen + 18
1   _pywrap_tensorflow.so         	0x00000001058ff490 perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 384
2   _pywrap_tensorflow.so         	0x000000010590000c perftools::gputools::internal::DsoLoader::GetLibcudaDsoHandle(void**) + 188
3   _pywrap_tensorflow.so         	0x0000000105901072 std::__1::__function::__func<tensorflow::Status (*)(void**), std::__1::allocator<tensorflow::Status (*)(void**)>, tensorflow::Status (void**)>::operator()(void**&&) + 18
4   _pywrap_tensorflow.so         	0x000000010590080f perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so         	0x0000000105900d6f perftools::gputools::internal::CachedDsoLoader::GetLibcudaDsoHandle() + 111
6   _pywrap_tensorflow.so         	0x0000000105982b85 perftools::gputools::initialize_cuda_gpu_executor() + 37
7   dyld                          	0x00007fff63346d4b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265
8   dyld                          	0x00007fff63346ed8 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                          	0x00007fff633438d1 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305
10  dyld                          	0x00007fff63343758 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 138
11  dyld                          	0x00007fff633439c9 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                          	0x00007fff633390ca dyld::runInitializers(ImageLoader*) + 89
13  dyld                          	0x00007fff63340231 dlopen + 578
14  libdyld.dylib                 	0x00007fff9005c857 dlopen + 59
15  org.python.python             	0x00000001000fc86f _PyImport_GetDynLoadFunc + 303
...

@peterswang btw, I have seen see this strlen + 18 crash when

DSO was is not in the library path
I was using El Capitan and disabled System Integrity Protection

There's a bunch of separate issues on this thread. The half_plus_two issue should have been resolved since. @tomnielsen @Velkan as for the wheel problem -- I haven't seen that other than when there's a lingering old version of setuptools around. Can you try installing in a virtualenv? Especially on Mac that often helps.

I also hit the same issue on Mac. I investigated a little bit.
As mentioned earlier in this thread, the crash happens because tensorflow couldn't load a cuda shared library. tensorflow shouldn't crash but it did because LOG(INFO) <<  ... << getenv("LD_LIBRARY_PATH") doesn't support char pointer being nullptr. I enclosed the relevant code at the end of this message.
As for why tensorflow didn't find the cuda shared library, in my case, it was because ipython from anaconda doesn't inherit the environment variable from my bash shell. I switched to use python instead of ipython, tensorflow was able to load the cuda shared library.
Hopefully this finding will help others who have similar problem.
tensorflow/stream_executor/dso_loader.cc@93
/* static */ port::Status DsoLoader::GetDsoHandle(port::StringPiece path,
                                                  void** dso_handle,
                                                  LoadKind load_kind) {

  int dynload_flags =
      RTLD_LAZY | (load_kind == LoadKind::kLocal ? RTLD_LOCAL : RTLD_GLOBAL);
  string path_string = path.ToString();
  *dso_handle = dlopen(path_string.c_str(), dynload_flags);
  if (*dso_handle == nullptr) {
    LOG(INFO) << "Couldn't open CUDA library " << path
              << ". LD_LIBRARY_PATH: " << getenv("LD_LIBRARY_PATH");
    return port::Status(
        port::error::FAILED_PRECONDITION,
        port::StrCat("could not dlopen DSO: ", path, "; dlerror: ", dlerror()));
  }
  LOG(INFO) << "successfully opened CUDA library " << path
            << (load_kind == LoadKind::kLocal ? " locally" : " globally");
  return port::Status::OK();
}

probably common.cc in protobuf
LogMessage& LogMessage::operator<<(const char* value) {
  message_ += value;
  return *this;
}

XCode c++ library string
...

_LIBCPP_INLINE_VISIBILITY basic_string& operator+=(const value_type* __s)         {return append(__s);}

...

basic_string<_CharT, _Traits, _Allocator>::append(const value_type* __s)
{
    _LIBCPP_ASSERT(__s != nullptr, "string::append received nullptr");
    return append(__s, traits_type::length(__s));
}

...

static inline size_t length(const char_type* __s) {return strlen(__s);}

@Invisibility nice digging. I've seen similar looking crashes on Mac in other places with similar stack trace, now I know where to look for. BTW, you can set env var in anaconda directly using something like this
import os
os.environ["LD_LIBRARY_PATH"]=....
import tensorflow

Thanks for the tip, @yaroslavvb. I tried, but it still crashed. It seems that for some reason dlopen in DsoLoader::GetDsoHandle didn't see the environment variable. I noticed that in my environment, ipython is a script with the following content:
#!/bin/bash <my anaconda virtual environment path>/bin/python.app
if __name__ == '__main__':
    import sys
    import IPython

    sys.exit(IPython.start_ipython())

My workaround is to avoid using python.app, i.e. running the following command insteead of directly running ipython:
python <my anaconda virtual environment path>/bin/ipython

I'm also getting a segmentation fault in python.
$ python test.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.7.5.dylib locally
[1]    34129 segmentation fault  python test.py

test.py
import tensorflow as tf

# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)

# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Runs the op.
print sess.run(c)

System Info:
Process:               Python [34031]
Path:                  /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               2.7.11 (2.7.11)
Code Type:             X86-64 (Native)
Parent Process:        zsh [33999]
Responsible:           iTerm2 [508]

OS Version:            Mac OS X 10.11.6 (15G31)
Kernel Version:	Darwin 15.6.0
Graphics: NVIDIA GeForce GTX 960, NVIDIA GeForce GTX 960, PCIe

Stack Trace
Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000

VM Regions Near 0:
-->
    __TEXT<PHONE_NUMBER>82c000-000000010782e000 [    8K] r-x/rwx SM=COW  /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib             	0x00007fff8ce51132 strlen + 18
1   _pywrap_tensorflow.so         	0x000000010a02b912 perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 402
2   _pywrap_tensorflow.so         	0x000000010a02c602 perftools::gputools::internal::DsoLoader::GetLibcudaDsoHandle(void**) + 258
3   _pywrap_tensorflow.so         	0x0000000109e44002 std::__1::__function::__func<tensorflow::Status (*)(tensorflow::shape_inference::InferenceContext*), std::__1::allocator<tensorflow::Status (*)(tensorflow::shape_inference::InferenceContext*)>, tensorflow::Status (tensorflow::shape_inference::InferenceContext*)>::operator()(tensorflow::shape_inference::InferenceContext*&&) + 18
4   _pywrap_tensorflow.so         	0x000000010a02ceef perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so         	0x000000010a02d450 perftools::gputools::internal::CachedDsoLoader::GetLibcudaDsoHandle() + 112
6   _pywrap_tensorflow.so         	0x000000010a0b0195 perftools::gputools::initialize_cuda_gpu_executor() + 37
7   dyld                          	0x00007fff6cb8710b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265
8   dyld                          	0x00007fff6cb87284 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                          	0x00007fff6cb838bd ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305
10  dyld                          	0x00007fff6cb83743 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 127
11  dyld                          	0x00007fff6cb839b3 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                          	0x00007fff6cb78eb0 dyld::runInitializers(ImageLoader*) + 89
13  dyld                          	0x00007fff6cb80308 dlopen + 555
14  libdyld.dylib                 	0x00007fff960d779c dlopen + 59
15  org.python.python             	0x00000001078e7892 _PyImport_GetDynLoadFunc + 309
...

Try Python3 instead of Python2

I'm currently using a GTX 960 2GB at 4096x2160@60Hz. Based on  https://github.com/tensorflow/tensorflow/issues/2840, I wonder if all the VRAM is used up for the display which is why I'm getting a segmentation fault?
I have a GTX 960 4GB that I can test tomorrow.

Previously I was following these instructions https://gist.github.com/Mistobaan/dd32287eeb6859c6668d and built from master. Now I used https://gist.github.com/ageitgey/819a51afa4613649bd18 and built from tag cuda_osx  and it worked. Note: I had to run git submodule update --init prior to bazel build commands and also prior to creating the pip packages cp -r bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/__main__/* bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/  from https://github.com/tensorflow/tensorflow/issues/2040#issuecomment-217833442 Notice the amount of free memory available!
python test.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.7.5.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 960
major: 5 minor: 2 memoryClockRate (GHz) 1.367
pciBusID 0000:04:00.0
Total memory: 2.00GiB
Free memory: 176.32MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:841] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0)
Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0
I tensorflow/core/common_runtime/direct_session.cc:175] Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0

MatMul: /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:818] MatMul: /job:localhost/replica:0/task:0/gpu:0
b: /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:818] b: /job:localhost/replica:0/task:0/gpu:0
a: /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:818] a: /job:localhost/replica:0/task:0/gpu:0
[[ 22.  28.]
 [ 49.  64.]]

I also rebuilt this using master branch and it worked. I think the key is TF_UNOFFICIAL_SETTING=1 ./configure

BTW, there's now a daily wheel with Mac GPU support:
export url=http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-mac-gpu/TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=gpu-mac/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.9.0-py3-none-any.whl
pip install --upgrade $url

There is a bug with loading libcuda.dylib - the default cuda install creates libcuda.dylib, but tensorflow tries to load libcuda.1.dylib .  This fails, resorting to using the LD_LIBRARY_PATH which if NULL crashes.  If you copy libcuda.dylib to libcuda.1.dylib it loads fine.

Thanks @alexcolburn, copying libcuda.dylib to libcuda.1.dylib indeed fixed this issue on my Mac 10.10

I confirmed comment of @xuanchien . So a simple solution. Worked on my MBP 2014 (nvidia 750M) with Mac OS 10.12

Thanks. I applied the same trick and now my TF on python 2.7 on OSX 10.11.5 works.

I'm closing this issue, I believe we have fixed the cuda version problem as well.

The issues still persist in tensorflow 1.0.0 alpha:
python -c "import tensorflow; print(tensorflow.__version__)" I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcublas.8.0.dylib locally I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcudnn.5.dylib locally I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcufft.8.0.dylib locally I tensorflow/stream_executor/dso_loader.cc:116] Couldn't open CUDA library libcuda.1.dylib. LD_LIBRARY_PATH: :/usr/local/cuda/lib I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: Olegs-MacBook-Pro-4.local I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: Invalid argument: expected %d.%d or %d.%d.%d form for driver version; got "" I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1065] LD_LIBRARY_PATH: :/usr/local/cuda/lib I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1066] failed to find libcuda.so on this system: Failed precondition: could not dlopen DSO: libcuda.1.dylib; dlerror: dlopen(libcuda.1.dylib, 6): image not found I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcurand.8.0.dylib locally 1.0.0-alpha
alexcolburn advice to link libcuda.1.dylib to libcuda.dylib does help.

Same problem after installing Tensorflow today. Version info:
$ python --version
Python 2.7.13
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Oct_30_22:18:43_CDT_2016
Cuda compilation tools, release 8.0, V8.0.54
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-8<IP_ADDRESS>)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Problem description:
$ python -c "import tensorflow"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcuda.1.dylib. LD_LIBRARY_PATH: /Developer/NVIDIA/CUDA-8.0/lib
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcuda.dylib. LD_LIBRARY_PATH: /Developer/NVIDIA/CUDA-8.0/lib
...

After creating a symlink:
$ cd /Developer/NVIDIA/CUDA-8.0/lib
$ sudo ln -s libcudart.8.0.dylib libcuda.1.dylib
$  python -c "import tensorflow"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 75, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/node_def_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 22, in <module>
    serialized_pb=_b('\n/tensorflow/core/framework/resource_handle.proto\x12\ntensorflow\"m\n\x0eResourceHandle\x12\x0e\n\x06\x64\x65vice\x18\x01 \x01(\t\x12\x11\n\tcontainer\x18\x02 \x01(\t\x12\x0c\n\x04name\x18\x03 \x01(\t\x12\x11\n\thash_code\x18\x04 \x01(\x04\x12\x17\n\x0fmaybe_type_name\x18\x05 \x01(\tB4\n\x18org.tensorflow.frameworkB\x13ResourceHandleProtoP\x01\xf8\x01\x01\x62\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'syntax'

The remaining error was due to protobuf not being installed correctly. I had previously run brew install protobuf. Once I unlinked it, Tensorflow works correctly (I'm not sure if this is breaking something else):
$ brew unlink protobuf
$ python -c "import tensorflow"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

This is still a common problem on OS X 10.11.5 / 10.11.6 as we have new team members following the instructions at https://www.tensorflow.org/install/install_mac
The solution is to create a link in /usr/local/cuda/lib with 'sudo ln -s /usr/local/cuda/lib/libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib'

@victorv Unfortunately, this problem still persists.
I am getting
(tensorflow) systems-mbp:~ test$ python
Python 3.5.2 |Anaconda 4.3.0 (x86_64)| (default, Jul  2 2016, 17:52:12)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
Segmentation fault: 11

Here's a successful 10.11.6 build. Mainly for my own documentation, it has been greatly simplified and Just Works™ :
Pre-requisites:
OS X 10.11.6
Python 2.7.11
pip 9.0.1
Download and install directly from nVIDIA:

Install CUDA-8.0
Install cuDNN-5.1
a. Copy files to /Developer/NVIDIA/CUDA-8.0/{lib,include}
b. Symlink ln -s /Developer/NVIDIA/CUDA-8.0lib/cudnn* /usr/local/cuda/lib/
c. Create symlink for libcuda.1.dylib
ln -s ln -s /usr/local/cuda/lib/libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib

Test CUDA install
Copy /Developer/NVIDIA/CUDA-8.0/samples to your working directory
cd $CUDA_SAMPLE_DIR/
a. To make all the samples: make -j<# of threads>
b. To make only the deviceQueryDrv Utility, cd to 1_Utilities/deviceQueryDrv and type make -j<# of threads>
$CUDA_SAMPLE_DIR/bin/x86_64/darwin/bin/x86_64/darwin/release/deviceQueryDrv
expected output:
» bin/x86_64/darwin/bin/x86_64/darwin/release/deviceQueryDrv
bin/x86_64/darwin/bin/x86_64/darwin/release/deviceQueryDrv Starting...

CUDA Device Query (Driver API) statically linked version
Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 960"
  CUDA Driver Version:                           8.0
  CUDA Capability Major/Minor version number:    5.2
  Total amount of global memory:                 4096 MBytes<PHONE_NUMBER> bytes)
  ( 8) Multiprocessors, (128) CUDA Cores/MP:     1024 CUDA Cores
  GPU Max Clock rate:                            1291 MHz (1.29 GHz)
  Memory Clock rate:                             3505 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 1048576 bytes
  Max Texture Dimension Sizes                    1D=(65536) 2D=(65536, 65536) 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size (x,y,z):<PHONE_NUMBER>, 65535, 65535)
  Texture alignment:                             512 bytes
  Maximum memory pitch:<PHONE_NUMBER> bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Result = PASS

Install Tensorflow

Automagically install GPU version pip install tensorflow-gpu

Test
tensorflow-test.py:
import tensorflow as tf

# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)

# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Runs the op.
print sess.run(c)

expected output:
» python ../tensorflow-test.py
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 960
major: 5 minor: 2 memoryClockRate (GHz) 1.291
pciBusID 0000:04:00.0
Total memory: 4.00GiB
Free memory: 921.46MiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0)
Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0
I tensorflow/core/common_runtime/direct_session.cc:255] Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0

MatMul: (MatMul): /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:827] MatMul: (MatMul)/job:localhost/replica:0/task:0/gpu:0
b: (Const): /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:827] b: (Const)/job:localhost/replica:0/task:0/gpu:0
a: (Const): /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:827] a: (Const)/job:localhost/replica:0/task:0/gpu:0
[[ 22.  28.]
 [ 49.  64.]]

@hbfs please clarify specifics of:
Install cuDNN-5.1
a. Copy files to /Developer/NVIDIA/CUDA-8.0/{lib,include}
b. Symlink ln -s /Developer/NVIDIA/CUDA-8.0lib/cudnn* /usr/local/cuda/lib/

What do the curly brackets mean?
Why do you have a directory CUDA-8.0lib

Had managed to build TF 0.12 by following How to fix dyld: Library not loaded: @rpath/libcudart.7.5.dylib issue when you build tensorflow on your Mac.
Just tried the building TF 1.0 from source using the same method and now the following error shows up after the patch in genrule-setup.sh:
ERROR: /Users/peter_wang/TF/tensorflow-1.0/tensorflow/python/BUILD:2279:1: Linking of rule '//tensorflow/python:_pywrap_tensorflow.so' failed: link_dynamic_library.sh failed: error executing command external/bazel_tools/tools/cpp/link_dynamic_library.sh no ignored ignored ignored external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -shared -o ... (remaining 648 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
clang: warning: argument unused during compilation: '-pthread'
ld: file not found: -lcudart.8.0
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Environment: 2012 MBP with Nvidia 650M, OS X 10.12.3, Python 2.7.13.
Links in /usr/local/cuda/lib, as suggested by @victorv was set before, though this doesn't seem to be the same issue.
lrwxr-xr-x  1 root  wheel     33 Feb 10 14:30 /usr/local/cuda/lib/libcuda.1.dylib -> /usr/local/cuda/lib/libcuda.dylib
-rwxr-xr-x  1 root  wheel  13504 Jan 24 11:58 /usr/local/cuda/lib/libcuda.dylib
lrwxr-xr-x@ 1 root  wheel     45 Nov  3 11:40 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-8.0/lib/libcudadevrt.a
lrwxr-xr-x@ 1 root  wheel     50 Nov  3 11:40 /usr/local/cuda/lib/libcudart.8.0.dylib -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.8.0.dylib
lrwxr-xr-x@ 1 root  wheel     46 Nov  3 11:40 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.dylib
lrwxr-xr-x@ 1 root  wheel     49 Nov  3 11:40 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart_static.a
"pip install tensorflow-gpu" does work. However, the standard build apparently doesn't include AVX & SSE, which I'd like to make use of as well.

@esd100 Do you have more detail on your segfault?

@victorv Tell me how to get more detail and I'll happily provide it.
I am using:
conda 4.3.11
Python 3.5.2 :: Anaconda 4.3.0 (x86_64)
=========================================
NVIDIA CUDA PATH
export PATH=/Developer/NVIDIA/CUDA-8.0/bin${PATH:+:${PATH}}
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-8.0/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
Files are installed in:
/Developer/NVIDIA/CUDA-8.0/
/Developer/NVIDIA/cuDNN
=========================================
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GT 750M"
CUDA Driver Version / Runtime Version          8.0 / 8.0
CUDA Capability Major/Minor version number:    3.0
Total amount of global memory:                 2048 MBytes<PHONE_NUMBER> bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP:     384 CUDA Cores
GPU Max Clock rate:                            926 MHz (0.93 GHz)
Memory Clock rate:                             2508 Mhz
Memory Bus Width:                              128-bit
L2 Cache Size:                                 262144 bytes
Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
Total amount of constant memory:               65536 bytes
Total amount of shared memory per block:       49152 bytes
Total number of registers available per block: 65536
Warp size:                                     32
Maximum number of threads per multiprocessor:  2048
Maximum number of threads per block:           1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size    (x,y,z):<PHONE_NUMBER>, 65535, 65535)
Maximum memory pitch:<PHONE_NUMBER> bytes
Texture alignment:                             512 bytes
Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
Run time limit on kernels:                     Yes
Integrated GPU sharing Host Memory:            No
Support host page-locked memory mapping:       Yes
Alignment requirement for Surfaces:            Yes
Device has ECC support:                        Disabled
Device supports Unified Addressing (UVA):      Yes
Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GT 750M
Result = PASS
=========================================
$ system_profiler SPSoftwareDataType)
System Version: macOS 10.12.2 (16C67)
Kernel Version: Darwin 16.3.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled
=========================================
$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: <IP_ADDRESS>.1.1480973914
=========================================
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-8<IP_ADDRESS>)
Target: x86_64-apple-darwin16.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

@esd100 {} is short hand for shell expansion (see 3.4.2 Brace expansion http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html)
8.0lib was a typo, missed a /, thanks for pointing that out. Fixed it.
@peterswang Yeah thanks for pointing that out. The pre-compiled version doesn't have all the optimization flags, for best performance and your system build from source. The tensorflow guide on that is a good one.

@esd100 Segmentation fault: 11 is usually followed by a OS X crash report.
You can find crash reports in your Console application -> User Diagnostic Reports
The segfault is usually a result of mixing your link libraries. You are using anaconda and have to follow the installation instructions to the letter to have anaconda work properly.
You can avoid lots of these install issues by using the docker images in caffe.

@victorv Thanks. Sorry for the delay.
Process:               python3.5 [25276]
Path:                  /anaconda/*/python3.5
Identifier:            python3.5
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        bash [25151]
Responsible:           python3.5 [25276]
User ID:               501

Date/Time:             2017-03-02 18:23:28.885 -0500
OS Version:            Mac OS X 10.12.3 (16D32)
Report Version:        12
Anonymous UUID:        0F0D56D5-52EB-7CFC-CD5E-ED3D20E9E476

Sleep/Wake UUID:       6CAAD8B9-E727-4612-A71C-06E5466B2E86

Time Awake Since Boot: 19000 seconds
Time Since Wake:       4200 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [0]

VM Regions Near 0:
-->
    __TEXT<PHONE_NUMBER>000000-0000000100001000 [    4K] r-x/rwx SM=COW  /anaconda/*/*.5

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib             	0x00007fffb545ab52 strlen + 18
1   _pywrap_tensorflow.so         	0x000000010537ad8a perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 282
2   _pywrap_tensorflow.so         	0x000000010537b69c perftools::gputools::internal::DsoLoader::GetCudnnDsoHandle(void**) + 204
3   _pywrap_tensorflow.so         	0x000000010537ccb2 std::__1::__function::__func<tensorflow::Status (*)(void**), std::__1::allocator<tensorflow::Status (*)(void**)>, tensorflow::Status (void**)>::operator()(void**&&) + 18
4   _pywrap_tensorflow.so         	0x000000010537c45f perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so         	0x000000010537c6ff perftools::gputools::internal::CachedDsoLoader::GetCudnnDsoHandle() + 111
6   _pywrap_tensorflow.so         	0x0000000105341e62 perftools::gputools::initialize_cudnn() + 242
7   dyld                          	0x00007fff6974e75f ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 385
8   dyld                          	0x00007fff6974e962 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                          	0x00007fff6974a1ee ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 338
10  dyld                          	0x00007fff69749268 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 138
11  dyld                          	0x00007fff697492fd ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                          	0x00007fff6973e70d dyld::runInitializers(ImageLoader*) + 87
13  dyld                          	0x00007fff69746150 dlopen + 556
14  libdyld.dylib                 	0x00007fffb5421a3e dlopen + 59
15  libpython3.5m.dylib           	0x00000001000f9f17 _PyImport_FindSharedFuncptr + 311
16  libpython3.5m.dylib           	0x00000001000db506 _PyImport_LoadDynamicModuleWithSpec + 422
17  libpython3.5m.dylib           	0x00000001000db0c2 _imp_create_dynamic + 258
18  libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
19  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
20  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
21  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
22  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
23  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
24  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
25  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
26  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
27  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
28  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
29  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
30  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
31  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
32  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
33  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
34  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
35  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
36  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
37  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
38  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
39  libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
40  libpython3.5m.dylib           	0x00000001000b52ab builtin_exec + 555
41  libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
42  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
43  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
44  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
45  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
46  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
47  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
48  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
49  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
50  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
51  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
52  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
53  libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
54  libpython3.5m.dylib           	0x000000010003430f function_call + 351
55  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
56  libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
57  libpython3.5m.dylib           	0x00000001000da3af PyImport_ImportModuleLevelObject + 1951
58  libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
59  libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
60  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
61  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
62  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
63  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
64  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
65  libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
66  libpython3.5m.dylib           	0x000000010003430f function_call + 351
67  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
68  libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
69  libpython3.5m.dylib           	0x00000001000da4f4 PyImport_ImportModuleLevelObject + 2276
70  libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
71  libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
72  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
73  libpython3.5m.dylib           	0x00000001000bbd2a PyEval_EvalFrameEx + 16874
74  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
75  libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
76  libpython3.5m.dylib           	0x00000001000b52ab builtin_exec + 555
77  libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
78  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
79  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
80  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
81  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
82  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
83  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
84  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
85  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
86  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
87  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
88  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
89  libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
90  libpython3.5m.dylib           	0x000000010003430f function_call + 351
91  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
92  libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
93  libpython3.5m.dylib           	0x00000001000da3af PyImport_ImportModuleLevelObject + 1951
94  libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
95  libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
96  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
97  libpython3.5m.dylib           	0x00000001000bbd2a PyEval_EvalFrameEx + 16874
98  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
99  libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
100 libpython3.5m.dylib           	0x00000001000b52ab builtin_exec + 555
101 libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
102 libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
103 libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
104 libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
105 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
106 libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
107 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
108 libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
109 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
110 libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
111 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
112 libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
113 libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
114 libpython3.5m.dylib           	0x000000010003430f function_call + 351
115 libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
116 libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
117 libpython3.5m.dylib           	0x00000001000da3af PyImport_ImportModuleLevelObject + 1951
118 libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
119 libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
120 libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
121 libpython3.5m.dylib           	0x00000001000bbd2a PyEval_EvalFrameEx + 16874
122 libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
123 libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
124 libpython3.5m.dylib           	0x00000001000e642c PyRun_InteractiveOneObject + 588
125 libpython3.5m.dylib           	0x00000001000e5d5e PyRun_InteractiveLoopFlags + 206
126 libpython3.5m.dylib           	0x00000001000e5c5c PyRun_AnyFileExFlags + 60
127 libpython3.5m.dylib           	0x00000001000fcd47 Py_Main + 3591
128 python                        	0x0000000100000dc7 main + 215
129 python                        	0x0000000100000ce4 start + 52

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x0000000000000000  rcx: 0x0000000000000000  rdx: 0x0000000000000000
  rdi: 0x0000000000000000  rsi: 0x0000000000000307  rbp: 0x00007fff5bff8e20  rsp: 0x00007fff5bff8e20
   r8: 0x00000000fffffff1   r9: 0x0000000101202778  r10: 0x000000000000000f  r11: 0x0000000000000000
  r12: 0x00007fff5bff9209  r13: 0xf7bee2bde4bf00e0  r14: 0x00007fff5bff9270  r15: 0x0000000000000010
  rip: 0x00007fffb545ab52  rfl: 0x0000000000010246  cr2: 0x0000000000000000

Logical CPU:     6
Error Code:      0x00000004
Trap Number:     14

Binary Images:
       0x100000000 -        0x100000ff7 +python (???) <A92DBDA7-44FF-35CF-8860-69BA977B417E> /anaconda/*/python
       0x100003000 -        0x1001b5fff +libpython3.5m.dylib (3.5) <D24D7ED3-029D-3C55-8431-04B5A2E8639A> /anaconda/*/libpython3.5m.dylib
       0x1003e6000 -        0x1003e7fff +_heapq.so (???) <01E9935D-050D-3732-BB3C-1F1A941DED4D> /anaconda/*/_heapq.so
       0x1003eb000 -        0x1003edfff +readline.so (???) <2BBC4898-3D25-3E58-BED2-4CA01365FFB9> /anaconda/*/readline.so
       0x1003f5000 -        0x1003f5ff7 +_opcode.so (???) <8F431933-E6A7-3318-81E4-C8157F1DA7F6> /anaconda/*/_opcode.so
       0x1003f9000 -        0x1003f9fff +grp.so (???) <75A30066-2CB2-3F6E-ADAC-B249233CDB42> /anaconda/*/grp.so
       0x1003fc000 -        0x1003fdfff +_random.so (???) <3326504F-74E4-33F3-8D0E-0468B6480468> /anaconda/*/_random.so
       0x100731000 -        0x100762fff +libreadline.6.2.dylib (6.2) <8B0801EE-2449-E9D6-850B-4E7FB34BF7FB> /anaconda/*/libreadline.6.2.dylib
       0x100778000 -        0x10078aff7 +_ctypes.so (???) <6784BCB6-23E1-3728-920D-D75D0AC4F34F> /anaconda/*/_ctypes.so
       0x10079b000 -        0x10079ffff +_struct.so (???) <42353FCD-910E-319F-8BD5-F4A5BD9AAF2E> /anaconda/*/_struct.so
       0x1007e8000 -        0x1007edff7 +math.so (???) <6301B65C-26E9-3CF7-9013-1239B15EF4D3> /anaconda/*/math.so
       0x1007f4000 -        0x1007f7ff7 +_lzma.so (???) <45350179-45D9-3E0C-9034-A863F5978360> /anaconda/*/_lzma.so
       0x101412000 -        0x1015a1fff +multiarray.cpython-35m-darwin.so (???) <C3082A47-3D27-3214-A3D3-346C8111EDE2> /anaconda/*/multiarray.cpython-35m-darwin.so
       0x10169f000 -        0x1016abfff +_datetime.so (???) <2264E487-677B-34D3-A546-9745721617A1> /anaconda/*/_datetime.so
       0x1016b6000 -        0x101789fff +umath.cpython-35m-darwin.so (???) <4F1ED886-054E-379D-96EE-5A522B1AD8FA> /anaconda/*/umath.cpython-35m-darwin.so
       0x1017c2000 -        0x1017d1fff +_pickle.so (???) <76465AD0-A0A7-311E-BDAE-09A7768E8860> /anaconda/*/_pickle.so
       0x1017dd000 -        0x1017f5ff7 +_bz2.so (???) <F29CB3C3-69BD-340E-AC38-BAA8C8F87D51> /anaconda/*/_bz2.so
       0x1017fb000 -        0x1017fcff7 +lapack_lite.cpython-35m-darwin.so (???) <EAAAB223-4E4A-3E1D-B0CE-E8A639265D3C> /anaconda/*/lapack_lite.cpython-35m-darwin.so
       0x102a80000 -        0x102ab4fff +liblzma.5.dylib (8.2) <DCF7E34A-AED3-3DB8-8DD8-F17522DAAE42> /anaconda/*/liblzma.5.dylib
       0x102afb000 -        0x102afdff7 +_hashlib.so (???) <2A1F4AD5-F0A5-3636-B93A-87A1158EEDB7> /anaconda/*/_hashlib.so
       0x102b02000 -        0x102b4cff7 +libssl.1.0.0.dylib (0) <9F42DCB5-702D-327C-84C1-AB64D6F942E1> /anaconda/*/libssl.1.0.0.dylib
       0x102b69000 -        0x102cea227 +libcrypto.1.0.0.dylib (0) <F7749773-18B9-33DE-B4BB-A9D1BFB6D96E> /anaconda/*/libcrypto.1.0.0.dylib
       0x102e20000 -        0x102e38fff +_umath_linalg.cpython-35m-darwin.so (???) <3DDEBEDD-5C80-339D-9770-6D62514FAA06> /anaconda/*/_umath_linalg.cpython-35m-darwin.so
       0x102ec6000 -        0x102ecfff7 +fftpack_lite.cpython-35m-darwin.so (???) <58E02D71-415E-36AA-9D54-3F32E4786163> /anaconda/*/fftpack_lite.cpython-35m-darwin.so
       0x103040000 -        0x1030eefff +mtrand.cpython-35m-darwin.so (???) <BE72B37C-5265-353B-AD9B-A44E9E57640D> /anaconda/*/mtrand.cpython-35m-darwin.so
       0x10321c000 -        0x10637d767 +_pywrap_tensorflow.so (0) <CFBCCB99-C59C-389E-85E0-DC9B3AE13CA8> /anaconda/*/_pywrap_tensorflow.so
       0x112a39000 -        0x112a74fff +libcudart.8.0.dylib (0) <6E55D67C-1A95-3C2B-B2C1-C3F5827A3F03> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.8.0.dylib
       0x113000000 -        0x11333aff7 +libcublas.8.0.dylib (0) <373C3010-E15D-3563-B655-76532180F4CF> /Developer/NVIDIA/CUDA-8.0/lib/libcublas.8.0.dylib
    0x7fff69739000 -     0x7fff69776267  dyld (421.2) <947FC440-80F9-32F7-A773-6FC418FE1AB7> /usr/lib/dyld
    0x7fff9c9d9000 -     0x7fff9c9d9fff  com.apple.Accelerate (1.11 - Accelerate 1.11) <D700DBDF-69AE-37A2-B9C7-0961CF0B6841> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    0x7fff9c9f2000 -     0x7fff9cf0bfeb  com.apple.vImage (8.1 - ???) <6408805B-67E9-3874-8D32-0BB814CE5CDA> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
    0x7fff9cf0c000 -     0x7fff9d07cff3  libBLAS.dylib (1185) <C7E42BBE-2337-3AEF-9C45-A2F2CB1A5B3E> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
    0x7fff9d07d000 -     0x7fff9d091ffb  libBNNS.dylib (14) <CFDEE88D-E002-347C-BC68-83099651585B> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
    0x7fff9d092000 -     0x7fff9d488fef  libLAPACK.dylib (1185) <2E8201CB-9A41-3D65-853E-841917FCE77B> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
    0x7fff9d489000 -     0x7fff9d49ffff  libLinearAlgebra.dylib (1185) <8CC29DE1-A231-3D5E-B5F1-DCC309036FE0> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
    0x7fff9d4a0000 -     0x7fff9d4a6fff  libQuadrature.dylib (3) <120F6228-A3D4-3184-89D7-785ADC2AC715> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
    0x7fff9d4a7000 -     0x7fff9d4bbff7  libSparseBLAS.dylib (1185) <C35235B7-CFA6-39A7-BD6E-79F4D9CAFD36> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
    0x7fff9d4bc000 -     0x7fff9d643fe7  libvDSP.dylib (600) <F59348AA-E1D3-3A27-8AB5-F546D38BFB76> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
    0x7fff9d644000 -     0x7fff9d6f6ffb  libvMisc.dylib (600) <70D4B548-47EE-3C6B-A93B-3EA6B60701E0> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
    0x7fff9d6f7000 -     0x7fff9d6f7fff  com.apple.Accelerate.vecLib (3.11 - vecLib 3.11) <A395B521-8E54-30F2-B4FE-355D68900DAF> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
    0x7fff9fe45000 -     0x7fffa02dffff  com.apple.CoreFoundation (6.9 - 1348.28) <A40AA224-7A50-3989-95D0-5A228A0E2FAF> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fffa1e27000 -     0x7fffa1ebcff7  com.apple.framework.IOKit (2.0.2 - 1324.30.13) <163BE7FA-B29A-348F-8B5F-E301F2E8C964> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
    0x7fffa634b000 -     0x7fffa666afff  com.apple.security (7.0 - 57740.31.2) <A47D7BAE-0591-3184-8E44-FB2EB08A19C2> /System/Library/Frameworks/Security.framework/Versions/A/Security
    0x7fffb3c42000 -     0x7fffb3c43ff3  libDiagnosticMessagesClient.dylib (102) <422911A4-E273-3E88-BFC4-DF6470E48242> /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fffb3e7b000 -     0x7fffb3e7bfff  libOpenScriptingUtil.dylib (172) <D025E180-BB3B-3FFA-98FC-B6835354D723> /usr/lib/libOpenScriptingUtil.dylib
    0x7fffb3e81000 -     0x7fffb3e82ff3  libSystem.B.dylib (1238) <9CB018AF-54E9-300F-82BE-81FE553C9154> /usr/lib/libSystem.B.dylib
    0x7fffb3f9a000 -     0x7fffb3f9aff3  libauto.dylib (187) <5BBF6A00-CC76-389D-84E7-CA88EDADE683> /usr/lib/libauto.dylib
    0x7fffb3f9b000 -     0x7fffb3fabff3  libbsm.0.dylib (34) <20084796-B04D-3B35-A003-EA11459557A9> /usr/lib/libbsm.0.dylib
    0x7fffb3fac000 -     0x7fffb3fbaff7  libbz2.1.0.dylib (38) <6FD3B63F-0F86-3A25-BD5B-E243F58792C9> /usr/lib/libbz2.1.0.dylib
    0x7fffb3fbb000 -     0x7fffb4011ff7  libc++.1.dylib (307.4) <BEE86868-F831-384C-919E-2B286ACFE87C> /usr/lib/libc++.1.dylib
    0x7fffb4012000 -     0x7fffb403cfff  libc++abi.dylib (307.2) <1CEF8ABB-7E6D-3C2F-8E0A-E7884478DD23> /usr/lib/libc++abi.dylib
    0x7fffb4064000 -     0x7fffb4064ff7  libcoretls.dylib (121.31.1) <BCC32537-4831-3E9F-876E-8C9F4CF52FD3> /usr/lib/libcoretls.dylib
    0x7fffb4065000 -     0x7fffb4066ff3  libcoretls_cfhelpers.dylib (121.31.1) <6F37C5AD-7999-3D31-A52F-7AEED935F32D> /usr/lib/libcoretls_cfhelpers.dylib
    0x7fffb4471000 -     0x7fffb4471fff  libenergytrace.dylib (15) <A1B040A2-7977-3097-9ADF-34FF181EB970> /usr/lib/libenergytrace.dylib
    0x7fffb457a000 -     0x7fffb479fffb  libicucore.A.dylib (57149.0.1) <6B5FDA93-AA88-318F-9608-C2A33D602EC7> /usr/lib/libicucore.A.dylib
    0x7fffb47a7000 -     0x7fffb47c0ffb  liblzma.5.dylib (10) <44BD0279-99DD-36B5-8A6E-C11432E2098D> /usr/lib/liblzma.5.dylib
    0x7fffb4a80000 -     0x7fffb4ab1ffb  libncurses.5.4.dylib (51.30.1) <A8C8F837-86A2-3EC2-B2D2-6E8A267847E4> /usr/lib/libncurses.5.4.dylib
    0x7fffb4b2c000 -     0x7fffb4efcd97  libobjc.A.dylib (706) <F9AFE665-A3A2-3285-9495-19803A565861> /usr/lib/libobjc.A.dylib
    0x7fffb4eff000 -     0x7fffb4f03fff  libpam.2.dylib (21.30.1) <71EB0D88-DE84-3C8D-A2C5-58AA282BC5BC> /usr/lib/libpam.2.dylib
    0x7fffb4fbe000 -     0x7fffb5106fe3  libsqlite3.dylib (253) <B5BA5C96-AB13-34A0-8237-DD52A0181DFE> /usr/lib/libsqlite3.dylib
    0x7fffb51fb000 -     0x7fffb5208fff  libxar.1.dylib (357) <58BFB84B-66FE-3299-AA3D-BBA178ADEE39> /usr/lib/libxar.1.dylib
    0x7fffb520c000 -     0x7fffb52fbffb  libxml2.2.dylib (30.11) <E12AF929-0FA5-3214-840F-C81E6AC9F36E> /usr/lib/libxml2.2.dylib
    0x7fffb5326000 -     0x7fffb5337ff3  libz.1.dylib (67) <46E3FFA2-4328-327A-8D34-A03E20BFFB8E> /usr/lib/libz.1.dylib
    0x7fffb5346000 -     0x7fffb534aff7  libcache.dylib (79) <0C8092D3-600F-3ADD-A036-F225B6CDCA43> /usr/lib/system/libcache.dylib
    0x7fffb534b000 -     0x7fffb5356ff7  libcommonCrypto.dylib (60092.30.2) <B16E29B6-EC8D-3A8F-9A89-DD9CF35F7C4B> /usr/lib/system/libcommonCrypto.dylib
    0x7fffb5357000 -     0x7fffb535efff  libcompiler_rt.dylib (62) <E992E8D9-037C-3454-A366-A25E4D31D6BB> /usr/lib/system/libcompiler_rt.dylib
    0x7fffb535f000 -     0x7fffb5367fff  libcopyfile.dylib (138) <64E285D9-5485-333B-AEE7-8B0C8FB9275F> /usr/lib/system/libcopyfile.dylib
    0x7fffb5368000 -     0x7fffb53ebfdf  libcorecrypto.dylib (442.30.20) <2074B932-FD79-30A9-8E90-AF25C49F2AF1> /usr/lib/system/libcorecrypto.dylib
    0x7fffb53ec000 -     0x7fffb541efff  libdispatch.dylib (703.30.5) <EA0CC14E-D559-3802-B4B2-0E8C7579AAC4> /usr/lib/system/libdispatch.dylib
    0x7fffb541f000 -     0x7fffb5424ff3  libdyld.dylib (421.2) <6F506653-FFF6-3DB8-84F1-109AE3C52F32> /usr/lib/system/libdyld.dylib
    0x7fffb5425000 -     0x7fffb5425ffb  libkeymgr.dylib (28) <1A318923-1200-3B06-B432-5007D82F195D> /usr/lib/system/libkeymgr.dylib
    0x7fffb5426000 -     0x7fffb5432ffb  libkxld.dylib (3789.41.3) <87550136-9353-348B-9CD9-C342B48C5AAF> /usr/lib/system/libkxld.dylib
    0x7fffb5433000 -     0x7fffb5433fff  liblaunch.dylib (972.30.7) <15FACC21-079A-3BDF-9AFB-4253EFDEB587> /usr/lib/system/liblaunch.dylib
    0x7fffb5434000 -     0x7fffb5439fff  libmacho.dylib (894) <A2F38EC1-C37C-3B93-B0E4-36B07C177F8C> /usr/lib/system/libmacho.dylib
    0x7fffb543a000 -     0x7fffb543cff3  libquarantine.dylib (85) <C1D7749F-5F5F-3BB9-BEFC-1F0B9DA941FD> /usr/lib/system/libquarantine.dylib
    0x7fffb543d000 -     0x7fffb543effb  libremovefile.dylib (45) <CD42974E-BE0B-39FC-9BFC-8A7540A04DC6> /usr/lib/system/libremovefile.dylib
    0x7fffb543f000 -     0x7fffb5457ff7  libsystem_asl.dylib (349.30.2) <EFAC72D7-CB13-3DF7-ADF3-EC6635C6F1EA> /usr/lib/system/libsystem_asl.dylib
    0x7fffb5458000 -     0x7fffb5458ff7  libsystem_blocks.dylib (67) <B8C3701D-5A91-3D35-999D-2DC8D5393525> /usr/lib/system/libsystem_blocks.dylib
    0x7fffb5459000 -     0x7fffb54e6fef  libsystem_c.dylib (1158.30.7) <2F881962-03CB-3B9D-A782-D98C1BBA4E3D> /usr/lib/system/libsystem_c.dylib
    0x7fffb54e7000 -     0x7fffb54eaffb  libsystem_configuration.dylib (888.30.2) <4FE3983C-E4ED-3939-A578-03AD29C99788> /usr/lib/system/libsystem_configuration.dylib
    0x7fffb54eb000 -     0x7fffb54eefff  libsystem_coreservices.dylib (41.4) <1A572B9E-0C47-320F-8C64-7990D0A5FB5A> /usr/lib/system/libsystem_coreservices.dylib
    0x7fffb54ef000 -     0x7fffb5507ff3  libsystem_coretls.dylib (121.31.1) <4676F06D-274D-31BE-B61C-4D7A4AEF4858> /usr/lib/system/libsystem_coretls.dylib
    0x7fffb5508000 -     0x7fffb550efff  libsystem_dnssd.dylib (765.30.11) <DC708D84-ED7D-3936-B996-A67C66B8DDAA> /usr/lib/system/libsystem_dnssd.dylib
    0x7fffb550f000 -     0x7fffb5538ff7  libsystem_info.dylib (503.30.1) <9ED9121C-F111-3FAD-BC2F-C95DEE1C9362> /usr/lib/system/libsystem_info.dylib
    0x7fffb5539000 -     0x7fffb555bff7  libsystem_kernel.dylib (3789.41.3) <B75B128C-7D7A-3318-91CD-82B5A69C5329> /usr/lib/system/libsystem_kernel.dylib
    0x7fffb555c000 -     0x7fffb55a3fe7  libsystem_m.dylib (3121.4) <266DB92B-A86F-3691-80FB-1B26AD73CFF3> /usr/lib/system/libsystem_m.dylib
    0x7fffb55a4000 -     0x7fffb55c2ff7  libsystem_malloc.dylib (116.30.3) <F40DEE3B-386A-3529-A3F7-98117ED55BF4> /usr/lib/system/libsystem_malloc.dylib
    0x7fffb55c3000 -     0x7fffb561affb  libsystem_network.dylib (856.30.16) <4AE368E9-605D-379D-B04C-2AC7455B8250> /usr/lib/system/libsystem_network.dylib
    0x7fffb561b000 -     0x7fffb5624ff3  libsystem_networkextension.dylib (563.30.15) <EB020B0C-7DF0-3EEF-8E3C-15DA3C01D687> /usr/lib/system/libsystem_networkextension.dylib
    0x7fffb5625000 -     0x7fffb562eff3  libsystem_notify.dylib (165.20.1) <E7FD3A7C-DD07-36E2-9FA4-7561F9F114DA> /usr/lib/system/libsystem_notify.dylib
    0x7fffb562f000 -     0x7fffb5637fe7  libsystem_platform.dylib (126.1.2) <3CA06D4E-C00A-36DE-AA65-3A390097D1F6> /usr/lib/system/libsystem_platform.dylib
    0x7fffb5638000 -     0x7fffb5642ff7  libsystem_pthread.dylib (218.30.1) <C869ED7C-BE29-3532-8E69-3A8DA1447EDC> /usr/lib/system/libsystem_pthread.dylib
    0x7fffb5643000 -     0x7fffb5646ff7  libsystem_sandbox.dylib (592.31.1) <7BBFDF96-293F-3DD9-B3A4-7C168280B441> /usr/lib/system/libsystem_sandbox.dylib
    0x7fffb5647000 -     0x7fffb5648fff  libsystem_secinit.dylib (24) <5C1F1E47-0F7D-3E25-8DEB-D9DB1F902281> /usr/lib/system/libsystem_secinit.dylib
    0x7fffb5649000 -     0x7fffb5650fff  libsystem_symptoms.dylib (532.30.6) <5D990CF5-B58F-39F7-B375-99B4EC62CFBD> /usr/lib/system/libsystem_symptoms.dylib
    0x7fffb5651000 -     0x7fffb5671ff7  libsystem_trace.dylib (518.30.7) <6D34D1EA-2A3C-3D2D-803E-A666E6AEEE52> /usr/lib/system/libsystem_trace.dylib
    0x7fffb5672000 -     0x7fffb5677ffb  libunwind.dylib (35.3) <9F7C2AD8-A9A7-3DE4-828D-B0F0F166AAA0> /usr/lib/system/libunwind.dylib
    0x7fffb5678000 -     0x7fffb56a1ff7  libxpc.dylib (972.30.7) <65E41BB6-EBD5-3D93-B0BE-B190CEE4DD93> /usr/lib/system/libxpc.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 12969
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=272.4M resident=0K(0%) swapped_out_or_unallocated=272.4M(100%)
Writable regions: Total=60.4M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=60.4M(100%)

                                VIRTUAL   REGION
REGION TYPE                        SIZE    COUNT (non-coalesced)
===========                     =======  =======
Kernel Alloc Once                    8K        2
MALLOC                            43.4M       24
MALLOC guard page                   16K        4
MALLOC_LARGE (reserved)            512K        3         reserved VM address space (unallocated)
VM_ALLOCATE                          8K        3
__DATA                            6396K      106
__LINKEDIT                       177.8M       31
__NV_CUDA                        169.4M        3
__TEXT                            94.6M       99
__UNICODE                          556K        2
__UNIXSTACK                       16.0M        2
shared memory                       12K        4
===========                     =======  =======
TOTAL                            508.6M      271
TOTAL, minus reserved VM space   508.1M      271

Have you tried setting LD_LIBRARY_PATH?
There's also this issue which tracked down similar "strlen + 18" to getenv LD_LIBRARY_PATH returning null
I actually see similar strlen + 18 with probability 50% on latest MacOS when I create session in GPU TensorFlow with CUDA_VISIBLE_DEVICES set to empty.
Most likely thing is that there's a piece of code which isn't checking for null pointers, which can happen non-deterministically.  For instance on MacOS, CFStringGetCStringPtr can return null non-deterministically. That output isn't checked for null and I fixed one such place in  https://github.com/tensorflow/tensorflow/pull/3448 but I see other return values that aren't checked for null, such as output of CFDictionaryGetValue in this line

@esd100 is could be LD_LIBRARY_PATH as @yaroslavvb has noted or you could be missing a cuda.1.dylib link in your /usr/local/cuda/lib directory.
lrwxr-xr-x   1 root  staff        33 Feb 16 12:33 libcuda.1.dylib -> /usr/local/cuda/lib/libcuda.dylib
ln -s libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib

@victorv @yaroslavvb I have the cuda.1.dylib link.
Would you recommend setting LD_LIBRARY_PATH in the bash profile? What do you set it to?

export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$DYLD_LIBRARY_PATH:$PATH

@yaroslavvb
Thanks for the input commands.
Some changes with that.
import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcudnn.5.dylib. LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib:/usr/local/cuda/lib:/Developer/NVIDIA/CUDA-8.0/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/daal/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/ipp/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/tbb/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/tbb/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/compiler/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/mkl/lib
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3517] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

I added
export DYLD_LIBRARY_PATH="/Developer/NVIDIA/cuDNN/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}"

and now ...
import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

First trial run ...
>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:874] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GT 750M
major: 3 minor: 0 memoryClockRate (GHz) 0.9255
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.85GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 750M, pci bus id: 0000:01:00.0)
>>> print(sess.run(hello))
b'Hello, TensorFlow!'

Do you know if this output means everything is running appropriately? I'm not sure why there are all the warning about the tensorflow compiling without the special instruction sets.  Any ideas?

Everything is fine. The warnings just warn you that things could be faster.

@martinwicke thanks!

@esd100 you can suppress info and warning log messages in the shell with export TF_CPP_MIN_LOG_LEVEL=2 or in python with os.environ['TF_CPP_MIN_LOG_LEVEL']="2"

For me (MacBook Pro macOS Sierra + CUDA-8.0)
export DYLD_LIBRARY_PATH="/Developer/NVIDIA/CUDA-8.0/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}"

Then
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

It works... THX @yaroslavvb @esd100

@esd100  @yazeedalrubyli @yaroslavvb
sorry to bother you, but I still met the same problem.
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcudnn.5.dylib. LD_LIBRARY_PATH: /Developer/NVIDIA/CUDA-8.0/lib:/Developer/NVIDIA/CUDA-8.0/lib :/Developer/NVIDIA/CUDA-8.0/lib :/Developer/NVIDIA/CUDA-8.0/lib ::/usr/local/cuda/lib:/usr/local/cuda/lib
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3517] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

Although I have added the environment path which is recommended by NVIDIA documentation and @esd100 .
vim ~/.bash_profile

export PATH=/Developer/NVIDIA/CUDA-8.0/bin${PATH:+:${PATH}}
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-8.0/lib\ ${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$DYLD_LIBRARY_PATH:$PATH

Any idea?...
Thank you very much!


Prevent invalidating table check constraints when altering table schema

Updated logic for dropping and renaming columns to ensure that no table checks are referenced.
Further improvements should eventually be made to ensure that table check constraints aren't modified by other schema alterations, such as modifying a column's type.

Closing this PR in favor of https://github.com/dolthub/go-mysql-server/pull/774 , now that the ga-format-pr commit has been pulled from this branch back into the original PR.


[Bug]: /var/lib/calico/ not mounted?

Expected Behavior
cattle-cluster-agent should deploy without issues
Actual Behavior
cattle-cluster-agent crashes and keeps on restarting throwing the error that /var/lib/calico is a non-existing directory (i assume not mounted to the pod?)
`~ $ kubectl get deployment cattle-cluster-agent -o yaml -n cattle-system
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"management.cattle.io/scale-available":"2"},"name":"cattle-cluster-agent","namespace":"cattle-system"},"spec":{"selector":{"matchLabels":{"app":"cattle-cluster-agent"}},"strategy":{"rollingUpdate":{"maxSurge":1,"maxUnavailable":0},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app":"cattle-cluster-agent"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"node-role.kubernetes.io/controlplane","operator":"In","values":["true"]}]},"weight":100},{"preference":{"matchExpressions":[{"key":"node-role.kubernetes.io/control-plane","operator":"In","values":["true"]}]},"weight":100},{"preference":{"matchExpressions":[{"key":"node-role.kubernetes.io/master","operator":"In","values":["true"]}]},"weight":100},{"preference":{"matchExpressions":[{"key":"cattle.io/cluster-agent","operator":"In","values":["true"]}]},"weight":1}],"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/os","operator":"NotIn","values":["windows"]}]}]}},"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["cattle-cluster-agent"]}]},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"env":[{"name":"CATTLE_FEATURES","value":"embedded-cluster-api=false,fleet=false,monitoringv1=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningv2=false,rke2=false"},{"name":"CATTLE_IS_RKE","value":"false"},{"name":"CATTLE_SERVER","value":"https://rancher.noobsecurity.labz"},{"name":"CATTLE_CA_CHECKSUM","value":"54a70c4edc11dcd7d1a4f09c95fd40981c026d303bf9dd6a9f51faae47b435a2"},{"name":"CATTLE_CLUSTER","value":"true"},{"name":"CATTLE_K8S_MANAGED","value":"true"},{"name":"CATTLE_CLUSTER_REGISTRY","value":""},{"name":"CATTLE_SERVER_VERSION","value":"v2.6.9"},{"name":"CATTLE_INSTALL_UUID","value":"f760c545-f9b0-4521-9913-7b8aee7ab4a2"},{"name":"CATTLE_INGRESS_IP_DOMAIN","value":"sslip.io"}],"image":"rancher/rancher-agent:v2.6.9","imagePullPolicy":"IfNotPresent","name":"cluster-register","volumeMounts":[{"mountPath":"/cattle-credentials","name":"cattle-credentials","readOnly":true}]}],"serviceAccountName":"cattle","tolerations":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/controlplane","value":"true"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane","operator":"Exists"},{"effect":"NoSchedule","key":"node-role.kubernetes.io/master","operator":"Exists"}],"volumes":[{"name":"cattle-credentials","secret":{"defaultMode":320,"secretName":"cattle-credentials-f60eae4"}}]}}}}
management.cattle.io/scale-available: "2"
creationTimestamp: "2022-11-14T10:21:59Z"
generation: 1
name: cattle-cluster-agent
namespace: cattle-system
resourceVersion: "1647"
uid: 8de5a12f-bc78-44a5-bac6-6972e5850bb0
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: cattle-cluster-agent
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: cattle-cluster-agent
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: node-role.kubernetes.io/controlplane
operator: In
values:
- "true"
weight: 100
- preference:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: In
values:
- "true"
weight: 100
- preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: In
values:
- "true"
weight: 100
- preference:
matchExpressions:
- key: cattle.io/cluster-agent
operator: In
values:
- "true"
weight: 1
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: NotIn
values:
- windows
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- cattle-cluster-agent
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- env:
- name: CATTLE_FEATURES
value: embedded-cluster-api=false,fleet=false,monitoringv1=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningv2=false,rke2=false
- name: CATTLE_IS_RKE
value: "false"
- name: CATTLE_SERVER
value: https://rancher.noobsecurity.labz
- name: CATTLE_CA_CHECKSUM
value: 54a70c4edc11dcd7d1a4f09c95fd40981c026d303bf9dd6a9f51faae47b435a2
- name: CATTLE_CLUSTER
value: "true"
- name: CATTLE_K8S_MANAGED
value: "true"
- name: CATTLE_CLUSTER_REGISTRY
- name: CATTLE_SERVER_VERSION
value: v2.6.9
- name: CATTLE_INSTALL_UUID
value: f760c545-f9b0-4521-9913-7b8aee7ab4a2
- name: CATTLE_INGRESS_IP_DOMAIN
value: sslip.io
image: rancher/rancher-agent:v2.6.9
imagePullPolicy: IfNotPresent
name: cluster-register
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /cattle-credentials
name: cattle-credentials
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: cattle
serviceAccountName: cattle
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/controlplane
value: "true"
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
volumes:
- name: cattle-credentials
secret:
defaultMode: 320
secretName: cattle-credentials-f60eae4
status:
conditions:

lastTransitionTime: "2022-11-14T10:21:59Z"
lastUpdateTime: "2022-11-14T10:29:48Z"
message: ReplicaSet "cattle-cluster-agent-5d6fc66cc6" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
lastTransitionTime: "2022-11-14T10:29:49Z"
lastUpdateTime: "2022-11-14T10:29:49Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
observedGeneration: 1
replicas: 1
unavailableReplicas: 1
updatedReplicas: 1`

Steps to Reproduce
Node OS: Ubuntu 22.04.01 LTS with latest updates (as of writing this post)
K8s: RKE2 v1.24.7+rke2r1
Rancher: 2.6.9
In rancher, go to Drivers -> Cluster Drivers -> select Linode LKE -> Activate
This sets the following:
Linode LKE
https://github.com/linode/kontainer-engine-driver-lke/releases/download/v0.0.6/kontainer-engine-driver-lke-linux-amd64
Note that v7 does not offer a driver download, just source code.
Proceed to create a new cluster based on Linode LKE.
Add any name.
Kept just the local admin.
Add linode access token. Click proceed.
region: eu-central
k8s version: 1.23
No tags
Node pool:
1 2GB Linode
1 4GB Linode
1 8GB Linode
Hit create.
Error Output
  Warning  FailedCreatePodSandBox  36m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "88bb11f0e21ed5abcb89cdd50e5b01c41dd556b7896de1b10bc84598c9785bff" network for pod "cattle-cluster-agent-5d6fc66cc6-h6r4p": networkPlugin cni failed to set up pod "cattle-cluster-agent-5d6fc66cc6-h6r4p_cattle-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/   Normal   SandboxChanged          36m (x12 over 36m)   kubelet            Pod sandbox changed, it will be killed and re-created.   Warning  FailedCreatePodSandBox  36m (x4 over 36m)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2f1f56cbb69cd205242468ff4461193c7edeb2db7ebe99519323e11886adc922" network for pod "cattle-cluster-agent-5d6fc66cc6-h6r4p": networkPlugin cni failed to set up pod "cattle-cluster-agent-5d6fc66cc6-h6r4p_cattle-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/   Warning  BackOff                 63s (x162 over 35m)  kubelet            Back-off restarting failed container

@d4b0zz Thanks for the report!
I've tested this issue locally against a Rancher instance (v2.6.9) using kontainer-engine-driver-lke v0.0.6 and my cattle-cluster-agent pods seem to be provisioning normally.
Just to make sure this wasn't a transient LKE error, could you try provisioning another cluster following the same steps?

@d4b0zz No worries! Glad you got things working 🙂


question about SAStest

1、how I can connect the EUD and test computer，LAN or USB.
2、is the test standards and methods are updated now?the version I used is <IP_ADDRESS>-2018-November-13.Not sure if it is the latest version。

Have you solved the above problems now?


Local npm install fails

Duplicate of https://github.com/greglobinski/gatsby-starter-hero-blog/issues/5. Please see the latest comments on this issue.