How to install Caffe on Mac OS X 10.11
Deep learning is a hot topic these days and it is greatly increased by the fact that AMD/nVidia video cards can be used for accelerating the training of very complex neural networks. In the meantime, some powerful frameworks have been developed by the community. One of the most known is Caffe which is specialized in image recognition, just one of the areas where I am more interested in.
However, it is not an easy task to get a working Caffe environment for a standard user. Of course, there are plenty of online documentation and some very nice guides but I didn’t find any for a Mac OS X 10.11.4 version working out of the box, so after struggling several hours trying to have everything ready, I decided to post here all the steps just in case it could help someone else.
In my case, I wanted to run the cool new automatic colorization methods using deep learning algorithms and the results are absolutely fantastic.
In addition, I also wanted to use CPU and GPU code paths in order to compare the processing speed, unfortunately, the video card I have on my workstation is an old AMD card and I need to have a nVidia one for using CUDA libraries so my only machine with that specs is a MacBook Pro with Mac OS X 10.11.4, 16GB RAM, Intel Core i7 4960HQ and a nVidia GeForce GT 750M (2GB).
Let’s start with the optional but highly recommended prerequisites:
- Check your Mac OS X and update everything is pending, you also need to install Xcode, 7.3 is the latest one but not supported by CUDA yet so later on, I will show you a work around for this issue
- Install Homebrew package manager and if possible, start up with a clean /usr/local directory
- Install Anaconda Python 2.7 on /usr/local directory. Once it is done, add to your ~/.bash_profile file:
export PATH=/usr/local/ananconda/bin:$PATH export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/anaconda:/usr/local/lib:/usr/lib
Now, we are installing the needed libraries:
- Install nVidia CUDA 7.5, in my case the exact version is 7.5.27. I also installed it on /usr/local/cuda. Once it is done, add to your ~/.bash_profile file:
export PATH=/usr/local/cuda/bin:$PATH export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_FALLBACK_LIBRARY_PATH
- Install nVidia cuDNN, you have to register and get approved by nVidia. I got 7.5 v5 but unfortunately, I was not able to use it, apparently some kind of incompatibility with my CUDA version or the Xcode used. In any case, once you have the tgz file, just uncompress it (tar -xfvz) and copy the contents of the /lib directory in to /usr/local/cuda/lib and cudnn.h to /usr/local/cuda/include directory.
- Install Intel MKL libraries, which is a part of Intel Parallel Studio XE Composer, you can download a trial version or a student/academic research one if you fulfill the requirements. MKL libraries are installed on /opt/intel/mkl. Once it is done, add to your ~/.bash_profile file:
- Using Homebrew, we are installing some needed libraries. At this point, it is very important to have the right environment variables, if you followed the guide step-by-step, you have them on your ~/.bash_profile but if you were working on the same terminal session, they are not loaded so, you can close this terminal and open a new one or just execute this command:
Before starting to download, compile and install the pending libraries, execute this command:
and you should see something like “Python 2.7.xx :: Anaconda 4.x.x (x86_64)”, if that is the case, the go on, otherwise, revise your environment vars as something is not well configured.
At this point, we are using the python provided by Anaconda so let’s execute these commands (note that in some cases, the GNU C/C++ 5.3.0 will be compiled and it takes a while, ~30min on my rig):
brew install --fresh -vd snappy leveldb gflags glog szip lmdb homebrew/science/opencv brew install --build-from-source --with-python --fresh -vd protobuf brew install --build-from-source --fresh -vd boost boost-python
- Finally, we can download Caffe, I am donwloading it to /usr/local/caffe:
git clone https://github.com/BVLC/caffe.git cd caffe cp Makefile.config.example Makefile.config
Next step is to configure it, open Makefile.config file with your preferred text editor (vim?) and let’s configure the following vars:
## Refer to http://caffe.berkeleyvision.org/installation.html # Contributions simplifying and improving our build system are welcome! # cuDNN acceleration switch (uncomment to build with cuDNN). # USE_CUDNN := 1 # CPU-only switch (uncomment to build without GPU support). # CPU_ONLY := 1 # uncomment to disable IO dependencies and corresponding data layers # USE_OPENCV := 0 # USE_LEVELDB := 0 # USE_LMDB := 0 # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary) # You should not set this flag if you will be reading LMDBs with any # possibility of simultaneous read and write # ALLOW_LMDB_NOLOCK := 1 # Uncomment if you're using OpenCV 3 # OPENCV_VERSION := 3 # To customize your choice of compiler, uncomment and set the following. # N.B. the default for Linux is g++ and the default for OSX is clang++ # CUSTOM_CXX := g++ # CUDA directory contains bin/ and lib/ directories that we need. CUDA_DIR := /usr/local/cuda # On Ubuntu 14.04, if cuda tools are installed via # "sudo apt-get install nvidia-cuda-toolkit" then use this instead: # CUDA_DIR := /usr # CUDA architecture setting: going with all of them. # For CUDA < 6.0, comment the *_50 lines for compatibility. CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \ -gencode arch=compute_20,code=sm_21 \ -gencode arch=compute_30,code=sm_30 \ -gencode arch=compute_35,code=sm_35 \ -gencode arch=compute_50,code=sm_50 \ -gencode arch=compute_50,code=compute_50 # BLAS choice: # atlas for ATLAS (default) # mkl for MKL # open for OpenBlas BLAS := mkl # Custom (MKL/ATLAS/OpenBLAS) include and lib directories. # Leave commented to accept the defaults for your choice of BLAS # (which should work)! BLAS_INCLUDE := /opt/intel/mkl/include BLAS_LIB := /opt/intel/mkl/lib # Homebrew puts openblas in a directory that is not on the standard search path # BLAS_INCLUDE := $(shell brew --prefix openblas)/include # BLAS_LIB := $(shell brew --prefix openblas)/lib # This is required only if you will compile the matlab interface. # MATLAB directory should contain the mex binary in /bin. # MATLAB_DIR := /usr/local # MATLAB_DIR := /Applications/MATLAB_R2012b.app # NOTE: this is required only if you will compile the python interface. # We need to be able to find Python.h and numpy/arrayobject.h. #PYTHON_INCLUDE := /usr/include/python2.7 \ # /usr/lib/python2.7/dist-packages/numpy/core/include # Anaconda Python distribution is quite popular. Include path: # Verify anaconda location, sometimes it's in root. ANACONDA_HOME := /usr/local/anaconda PYTHON_INCLUDE := $(ANACONDA_HOME)/include \ $(ANACONDA_HOME)/include/python2.7 \ $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \ # Uncomment to use Python 3 (default is Python 2) # PYTHON_LIBRARIES := boost_python3 python3.5m # PYTHON_INCLUDE := /usr/include/python3.5m \ # /usr/lib/python3.5/dist-packages/numpy/core/include # We need to be able to find libpythonX.X.so or .dylib. #PYTHON_LIB := /usr/lib PYTHON_LIB := $(ANACONDA_HOME)/lib # PYTHON_LIB := /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/ # Homebrew installs numpy in a non standard path (keg only) PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include PYTHON_LIB += $(shell brew --prefix numpy)/lib # Uncomment to support layers written in Python (will link against Python libs) # WITH_PYTHON_LAYER := 1 # Whatever else you find you need goes here. INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies INCLUDE_DIRS += $(shell brew --prefix)/include LIBRARY_DIRS += $(shell brew --prefix)/lib # Uncomment to use `pkg-config` to specify OpenCV library paths. # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.) # USE_PKG_CONFIG := 1 BUILD_DIR := build DISTRIBUTE_DIR := distribute # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171 # DEBUG := 1 # The ID of the GPU that 'make runtest' will use to run unit tests. TEST_GPUID := 0 # enable pretty build (comment to see full commands) Q ?= @
Update 23/August/2016: Instead of providing a partial Makefile.config file, a full copy of this file is attached above.
Give it a try…:
make clean make all -j8
If it compiles fine, you can skip this paragraph, you probably have Xcode 7.0.2 or a lower version which is OK…
but if all you get is this error:
“nvcc fatal : The version (‘70300’) of the host compiler (‘Apple clang’) is not supported”
it means you have the Xcode 7.3 and CUDA still does not support it.
Of course, it has a solution, to install a supported Xcode version and use it for compiling Caffe:
– go to https://developer.apple.com/downloads/ and log in
– download Command Line Tools OS X 10.11 Xcode 7.2 and install it
– now, switch to use this version:
sudo xcode-select --switch /Library/Developer/CommandLineTools
Remember that using:
sudo xcode-select -r
will configure the default compiler, which, in my case is 7.3.
Let’s try to compile it again:
make clean make all -j8
This time, it should be OK. Now, let’s compile the tests:
make test -j8
Everything went fine? Perfect! At this point, you probably want to run the tests but… they will fail(!). This time the guilty one is Apple and the latest OS X 10.11 “El Capitan” version, apparently it is undocumented but spending some time ‘googling’, I found that other application are failing too as it seems that Apple unset all DYLD_ prefix environment variables when running scripts. I only know one solution:
– to disable System Integrity Protection, which I totally do not recommend to do, or if you do that, revert to SIP on as soon as you check that all the tests were OK.
So, we can discard to run the tests and trust that everything is fine and continue with only a few more steps.
- Compiling PyCaffe, which is the python interface to Caffe:
for req in $(cat python/requirements.txt); do pip install $req; done make pycaffe make distribute
If everything went fine, you are almost there! Add this environment variable to your ~/.bash_profile:
And now the final test:
cd /usr/local/caffe/python python import caffe
You should get the python prompt and no errors at all which means that your python interface to Caffe is fully working and ready for some cool applications:
Python 2.7.11 |Anaconda 4.0.0 (x86_64)| (default, Dec 6 2015, 18:57:58) [GCC 4.2.1 (Apple Inc. build 5577)] on darwin Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> import caffe >>>
Update: if you receive some errors regarding to locale, you just need to add these new environment vars to your famous ~/.bash_profile file:
export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8
You are now ready for playing with some examples:
As you can see, it is not an easy task to configure Caffe environment but finally I got it and can test the Automatic Colorization algorithms for automatically converting grey-pictures into fully colored ones, in a few days I will write down a small post with my result.
Have fun with this great but a bit difficult to configure deep learning framework!!
i’m using Mac OSX 10.11
and followed your step one by one…
what i came to this point :
i get this error :
Mors-MBP:caffe MorM$ make all -j8
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
clang: warning: argument unused during compilation: ‘-pthread’
ld: library not found for -lcudnn
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] Error 1
Can you please help me? i’ve tried to install the x-code command line tool like you mention.. it’s still stuck there…
CUDA 7.5.27 (the one I used and the latest one) is not supported on x-code 7.3.x so you have to install the “Command Line Tools OS X 10.11 Xcode 7.2”, so basically, you can go to this step on the guide and change the compiler to 7.2, then, the compilation will work. Anyway, if you have any other issue, please let me know.
I did that – do u have any other idea ?
in other forums they suggested that maybe Mac can not read “libcaffe.so.1.0.0-rc3 “library – only the ones with .dylib but i don’t know what thats mean.
reading your original reply I can see:
“ld: library not found for -lcudnn”
so it seems you are trying to use cuDNN, something I tried but had to disable due to incompatibilities.
You just have to disable the cuDNN usage on the Makefile.config file (comment USE_CUDNN := 1 line).
In any case, instead of attaching a partial copy of the Makefile.config on this guide, I have updated it with the full content of the file, after the issues you were facing, I think is better.
Hope this fix help you.
[…] How to install Caffe on Mac OS X 10.11 […]
ython 2.7.12 |Anaconda 4.2.0 (x86_64)| (default, Jul 2 2016, 17:43:17)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import caffe
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/caffe/python/caffe/__init__.py”, line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File “/usr/local/caffe/python/caffe/pycaffe.py”, line 13, in
from ._caffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, \
ImportError: dlopen(/usr/local/caffe/python/caffe/_caffe.so, 2): Library not loaded: @rpath/libcaffe.so.1.0.0-rc3
Referenced from: /usr/local/caffe/python/caffe/_caffe.so
Reason: image not found
sorry ,I forget copy the whole caffe to the /usr/local/. then i do it and do it again just all right!
BTW, if in the end , your python has fail import cv2 you should add this to your .bash_profile
Excellent tutorial! Congratulations! After spending a day trying to install Caffe, I can say that it’s the best tutorial to install Caffe in Mac Os X “The Captain”!
Thank you for this awesome tutorial!
But I got an error after i entered make runtest :
dyld: Library not loaded: @rpath/libmkl_rt.dylib
Referenced from: /Users/username/caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5
Could you please help me to fix this problem?
[…] How to install Caffe on Mac OS X 10.11 […]
nvcc warning : The ‘compute_20’, ‘sm_20’, and ‘sm_21’ architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
I am getting the above warning.
What can I do about it ?
thanks for your time.
I have the same error and I Google this. Internet said this WARNING doesn’t prevent your Caffe from running. So you should be fine. By the way, do you have error while running “make pycaffe”? I have a fatal error saying ‘Python.h’ file not found
Hi, first of all. Thanks for the post. It helps a lot.
I do have some error during installation and I need help.
System: 10.12.1 (2015 Model with i5 CPU)
cuDNN (I don’t have a NVIDIA graphic but I installed this anyways)
I have “cuda”, “caffe” and “anaconda” under /usr/local
and I follow your instruction adding path and did run source to update batch profile
I have 11 or 12 warnings while running “make all” and “make test”
Most importantly, I have a Fetal Error while running “make pycaffe”
python/caffe/_caffe.cpp:1:10: fatal error: ‘Python.h’ file not found
I Google this and the internet told me to comment out the following
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python2.7 \
# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
But if that is the case, I believe I can’t use Anaconda’s version of Python
Can you help??
[…] http://www.megastormsystems.com/news/how-to-install-caffe-on-mac-os-x-10-11 […]
Hello, I am getting this error on my Mac 10.13.4 although I have given the path of installed boost_python
ld: library not found for -lboost_python
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [python/caffe/_caffe.so] Error 1
See here the path is given for the conda version of boost-python and as well as the general version of boost-python:
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/local/Cellar/lmdb/0.9.22/include /usr/local/Cellar/opencv/3.4.1_5/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/local/Cellar/lmdb/0.9.22/lib /usr/local/Cellar/opencv/3.4.1_5/lib /Users/tmondal/anaconda2/pkgs/boost-python-1.55.0-0/lib /usr/local/Cellar/boost-python/1.67.0/lib
Then also I am facing this error. Please help