Development blog

CRM64Pro Beta and DPF2XML updated

A minor revision of CRM64Pro GDK with some improvements is available for downloading. As it is a beta version, only Windows 64 bits DLL version is provided.

DPF2XML is also greatly improved and now is able to export/import all DPF resources to a XML file (MegaStorm Interchange XML v1.0 format or MIXv1). This MIXv1 file can be imported using EditorC64 so now, I have a way of importing all DPF into new CRM64Pro CDC format.

Next step is the release of EditorC64.


CRM64Pro v0.94 (Beta) released

And finally, the day of a beta release arrived.
As of now, only two versions are released, one for Windows 32bits and the other one for Windows 64bits. Both using Visual Studio 2015 and DLLs. In addition, an almost-finished documentation is also available. The rest of the packages will come later on.

You can find it at CRM64Pro GDK page.

I am working on adding the feature of importing DPF (from CRM32Pro) on EditorC64 in order to help on the project migrations from C32 to the new C64.

Any issue, doubt or comment you have, please let me know :).

P.S.: there is not any example of source code using it so if requested, I will prepare some small ones.


CRM64Pro current state

From time to time, I tend to forget the amount of work, all the invested hours and all the effort required for finishing any piece of code. Even a small game or application, if doing it right, will take a lot of hours, days and even weeks.

Developing something like CRM64Pro is not a less requiring time task than previous examples, in fact, it is a titanic task just for one person…

I forgot how hard was developing, till the last shape, CRM32Pro and creating a new beast with all the CRM32Pro features plus much more, all of them from the scratch, have taken me about 3 years.

It is true that during several months, due to my current job and/or personal things, I could not do anything but wait for better days. If you are interested, I have attached a screenshot of the number of commits performed from the beginning to now where you can see the development time line.

Finally and after investing soo much time on it, I am quite confident the result is a great cross-platform library for developing 2D games, greater than CRM32Pro although the timing is not the same. When CRM32Pro was launched, there was a lot of interest on developing 2D games, nowadays, the world is fully commited to 3D games but I think there is still place for good 2D titles so I hope all my efforts can help, at least, to the same number of people that the CRM32Pro did.

Of course, do you know? there are some pending tasks (check current status here), but the major components are finished. As some of you are asking me for a pre-release, I will do it during next days while I keep working on the Tile Engine.

Once current roadmap for CRM64Pro is done, I will focus on porting TitanBomber from C32 to C64, I am longing for seeing the bomberman-like clone running on Android devices!! and then I will go on with HX project

C64-Timeline


How to install Caffe on Mac OS X 10.11

Deep learning is a hot topic these days and it is greatly increased by the fact that AMD/nVidia video cards can be used for accelerating the training of very complex neural networks. In the meantime, some powerful frameworks have been developed by the community. One of the most known is Caffe which is specialized in image recognition, just one of the areas where I am more interested in.

However, it is not an easy task to get a working Caffe environment for a standard user. Of course, there are plenty of online documentation and some very nice guides but I didn’t find any for a Mac OS X 10.11.4 version working out of the box, so after struggling several hours trying to have everything ready, I decided to post here all the steps just in case it could help someone else.

In my case, I wanted to run the cool new automatic colorization methods using deep learning algorithms and the results are absolutely fantastic.

In addition, I also wanted to use CPU and GPU code paths in order to compare the processing speed, unfortunately, the video card I have on my workstation is an old AMD card and I need to have a nVidia one for using CUDA libraries so my only machine with that specs is a MacBook Pro with Mac OS X 10.11.4, 16GB RAM, Intel Core i7 4960HQ and a nVidia GeForce GT 750M (2GB).

Let’s start with the optional but highly recommended prerequisites:

  • Check your Mac OS X and update everything is pending, you also need to install Xcode, 7.3 is the latest one but not supported by CUDA yet so later on, I will show you a work around for this issue
  • Install Homebrew package manager and if possible, start up with a clean /usr/local directory
  • Install Anaconda Python 2.7 on /usr/local directory. Once it is done, add to your ~/.bash_profile file:
     
    export PATH=/usr/local/ananconda/bin:$PATH 
    export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/anaconda:/usr/local/lib:/usr/lib
    

Now, we are installing the needed libraries:

  • Install nVidia CUDA 7.5, in my case the exact version is 7.5.27. I also installed it on /usr/local/cuda. Once it is done, add to your ~/.bash_profile file:
    export PATH=/usr/local/cuda/bin:$PATH
    export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH
    export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_FALLBACK_LIBRARY_PATH
    
  • Install nVidia cuDNN, you have to register and get approved by nVidia. I got 7.5 v5 but unfortunately, I was not able to use it, apparently some kind of incompatibility with my CUDA version or the Xcode used. In any case, once you have the tgz file, just uncompress it (tar -xfvz) and copy the contents of the /lib directory in to /usr/local/cuda/lib and cudnn.h to /usr/local/cuda/include directory.
  • Install Intel MKL libraries, which is a part of Intel Parallel Studio XE Composer, you can download a trial version or a student/academic research one if you fulfill the requirements. MKL libraries are installed on /opt/intel/mkl. Once it is done, add to your ~/.bash_profile file:
    export DYLD_FALLBACK_LIBRARY_PATH=/opt/intel/mkl/lib:$DYLD_FALLBACK_LIBRARY_PATH
    
  • Using Homebrew, we are installing some needed libraries. At this point, it is very important to have the right environment variables, if you followed the guide step-by-step, you have them on your ~/.bash_profile but if you were working on the same terminal session, they are not loaded so, you can close this terminal and open a new one or just execute this command:
    source ~/.bash_profile
    

    Before starting to download, compile and install the pending libraries, execute this command:

    python --version
    

    and you should see something like “Python 2.7.xx :: Anaconda 4.x.x (x86_64)”, if that is the case, the go on, otherwise, revise your environment vars as something is not well configured.

    At this point, we are using the python provided by Anaconda so let’s execute these commands (note that in some cases, the GNU C/C++ 5.3.0 will be compiled and it takes a while, ~30min on my rig):

    brew install --fresh -vd snappy leveldb gflags glog szip lmdb homebrew/science/opencv
    brew install --build-from-source --with-python --fresh -vd protobuf
    brew install --build-from-source --fresh -vd boost boost-python
    
  • Finally, we can download Caffe, I am donwloading it to /usr/local/caffe:
    git clone https://github.com/BVLC/caffe.git
    cd caffe
    cp Makefile.config.example Makefile.config
    

    Next step is to configure it, open Makefile.config file with your preferred text editor (vim?) and let’s configure the following vars:

     ## Refer to http://caffe.berkeleyvision.org/installation.html
    # Contributions simplifying and improving our build system are welcome!
    
    # cuDNN acceleration switch (uncomment to build with cuDNN).
    # USE_CUDNN := 1
    
    # CPU-only switch (uncomment to build without GPU support).
    # CPU_ONLY := 1
    
    # uncomment to disable IO dependencies and corresponding data layers
    # USE_OPENCV := 0
    # USE_LEVELDB := 0
    # USE_LMDB := 0
    
    # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
    #    You should not set this flag if you will be reading LMDBs with any
    #    possibility of simultaneous read and write
    # ALLOW_LMDB_NOLOCK := 1
    
    # Uncomment if you're using OpenCV 3
    # OPENCV_VERSION := 3
    
    # To customize your choice of compiler, uncomment and set the following.
    # N.B. the default for Linux is g++ and the default for OSX is clang++
    # CUSTOM_CXX := g++
    
    # CUDA directory contains bin/ and lib/ directories that we need.
    CUDA_DIR := /usr/local/cuda
    # On Ubuntu 14.04, if cuda tools are installed via
    # "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
    # CUDA_DIR := /usr
    
    # CUDA architecture setting: going with all of them.
    # For CUDA < 6.0, comment the *_50 lines for compatibility.
    CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
    -gencode arch=compute_20,code=sm_21 \
    -gencode arch=compute_30,code=sm_30 \
    -gencode arch=compute_35,code=sm_35 \
    -gencode arch=compute_50,code=sm_50 \
    -gencode arch=compute_50,code=compute_50
    
    # BLAS choice:
    # atlas for ATLAS (default)
    # mkl for MKL
    # open for OpenBlas
    BLAS := mkl
    # Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
    # Leave commented to accept the defaults for your choice of BLAS
    # (which should work)!
    BLAS_INCLUDE := /opt/intel/mkl/include
    BLAS_LIB := /opt/intel/mkl/lib
    
    # Homebrew puts openblas in a directory that is not on the standard search path
    # BLAS_INCLUDE := $(shell brew --prefix openblas)/include
    # BLAS_LIB := $(shell brew --prefix openblas)/lib
    
    # This is required only if you will compile the matlab interface.
    # MATLAB directory should contain the mex binary in /bin.
    # MATLAB_DIR := /usr/local
    # MATLAB_DIR := /Applications/MATLAB_R2012b.app
    
    # NOTE: this is required only if you will compile the python interface.
    # We need to be able to find Python.h and numpy/arrayobject.h.
    #PYTHON_INCLUDE := /usr/include/python2.7 \
    #        /usr/lib/python2.7/dist-packages/numpy/core/include
    # Anaconda Python distribution is quite popular. Include path:
    # Verify anaconda location, sometimes it's in root.
    ANACONDA_HOME := /usr/local/anaconda
    PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
    $(ANACONDA_HOME)/include/python2.7 \
    $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
    
    # Uncomment to use Python 3 (default is Python 2)
    # PYTHON_LIBRARIES := boost_python3 python3.5m
    # PYTHON_INCLUDE := /usr/include/python3.5m \
    #                 /usr/lib/python3.5/dist-packages/numpy/core/include
    
    # We need to be able to find libpythonX.X.so or .dylib.
    #PYTHON_LIB := /usr/lib
    PYTHON_LIB := $(ANACONDA_HOME)/lib
    # PYTHON_LIB := /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/
    
    # Homebrew installs numpy in a non standard path (keg only)
    PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
    PYTHON_LIB += $(shell brew --prefix numpy)/lib
    
    # Uncomment to support layers written in Python (will link against Python libs)
    # WITH_PYTHON_LAYER := 1
    
    # Whatever else you find you need goes here.
    INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
    LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
    
    # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
    INCLUDE_DIRS += $(shell brew --prefix)/include
    LIBRARY_DIRS += $(shell brew --prefix)/lib
    
    # Uncomment to use `pkg-config` to specify OpenCV library paths.
    # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
    # USE_PKG_CONFIG := 1
    
    BUILD_DIR := build
    DISTRIBUTE_DIR := distribute
    
    # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
    # DEBUG := 1
    
    # The ID of the GPU that 'make runtest' will use to run unit tests.
    TEST_GPUID := 0
    
    # enable pretty build (comment to see full commands)
    Q ?= @
    

    Update 23/August/2016: Instead of providing a partial Makefile.config file, a full copy of this file is attached above.

    Give it a try…:

    make clean
    make all -j8
    

    If it compiles fine, you can skip this paragraph, you probably have Xcode 7.0.2 or a lower version which is OK…
    but if all you get is this error:
    “nvcc fatal : The version (‘70300’) of the host compiler (‘Apple clang’) is not supported”
    it means you have the Xcode 7.3 and CUDA still does not support it.
    Of course, it has a solution, to install a supported Xcode version and use it for compiling Caffe:
    – go to https://developer.apple.com/downloads/ and log in
    – download Command Line Tools OS X 10.11 Xcode 7.2 and install it
    – now, switch to use this version:

     sudo xcode-select --switch /Library/Developer/CommandLineTools 

    Remember that using:

     sudo xcode-select -r 

    will configure the default compiler, which, in my case is 7.3.
    Let’s try to compile it again:

    make clean
    make all -j8
    

    This time, it should be OK. Now, let’s compile the tests:

    make test -j8
    

    Everything went fine? Perfect! At this point, you probably want to run the tests but… they will fail(!). This time the guilty one is Apple and the latest OS X 10.11 “El Capitan” version, apparently it is undocumented but spending some time ‘googling’, I found that other application are failing too as it seems that Apple unset all DYLD_ prefix environment variables when running scripts. I only know one solution:
    – to disable System Integrity Protection, which I totally do not recommend to do, or if you do that, revert to SIP on as soon as you check that all the tests were OK.
    So, we can discard to run the tests and trust that everything is fine and continue with only a few more steps.

  • Compiling PyCaffe, which is the python interface to Caffe:
    for req in $(cat python/requirements.txt); do pip install $req; done
    make pycaffe
    make distribute
    

    If everything went fine, you are almost there! Add this environment variable to your ~/.bash_profile:

    export PYTHONPATH=/usr/local/caffe/python
    

    And now the final test:

    cd /usr/local/caffe/python
    python
    import caffe
    

    You should get the python prompt and no errors at all which means that your python interface to Caffe is fully working and ready for some cool applications:

    Python 2.7.11 |Anaconda 4.0.0 (x86_64)| (default, Dec  6 2015, 18:57:58) 
    [GCC 4.2.1 (Apple Inc. build 5577)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    >>> import caffe
    >>> 
    

    Update: if you receive some errors regarding to locale, you just need to add these new environment vars to your famous  ~/.bash_profile file:

    export LC_ALL=en_US.UTF-8
    export LANG=en_US.UTF-8
    

You are now ready for playing with some examples:

As you can see, it is not an easy task to configure Caffe environment but finally I got it and can test the Automatic Colorization algorithms for automatically converting grey-pictures into fully colored ones, in a few days I will write down a small post with my result.

Have fun with this great but a bit difficult to configure deep learning framework!!


YASM integration with Visual Studio 2010, 2012 and 2013

YASM is a great assembly compiler, it supports a lot of instruction sets, assembler syntaxes, output formats, … so since Visual Studio started to use the compiler intrinsics for some versions, in my case, for x64 code, I switched to use YASM as I don’t want to mind about different syntaxes or weird and crazy compiler keywords, I just want to create a single .asm file and rule all platforms!

As if all this was not enough, it can be integrated with Visual Studio, just unzip the package vsyasm and follow a few steps given in a readme.txt.

For the 2008 version, it worked fine but recently, I tried with Visual Studio 2010 and the latest YASM v1.3.0 but all I got, where error messages. Time for investigating it!

I checked the command line used by Visual Studio for calling YASM and quickly saw it was a problem with a parameter. YASM expected the parameter “-f” to be win32 or x64 but in newer Visual Studio versions (2010, 2012 and 2013), Win32 is used instead win32 so it caused the error.

I spent some time “googling” for it and I found that the issue was already reported and fixed by a patch on 2014-October-05:

So why it was failing then? Simple, the binary package I downloaded from the official YASM web site was compiled on 2014-August but the issue was fixed some months later.

At this point, I didn’t spend much time looking for a vsyasm binary compiled after 2014-October, I just went to the Git repository and downloaded a full copy of the source code (as of 2015-June-09). I thought it was very straightforward to compile it but well, sometimes I’m wrong :). It took me like 1 hour till I could build YASM with Visual Studio 2010. Finally, I got two packages, one for 32bits and other for 32/64bits, so I used them on my project and bingo! they worked fine as the patch promised!

I imagine that sooner than later, I new build will be available on the official web site that using recent code, will fix this and other bugs but in the meantime, I think it can help you (saving your time trying to compile it!) if I upload them ready to be used: