×

Blog

Doing Machine Vision on Google Coral with OpenCV

Petar Jalušić on 27 Jan 2020

Doing Machine Vision on Google Coral with OpenCV

While developing computer vision applications for embedded devices, I realized that execution time is going to be the number one priority. However, after failing to install all required libraries for such a task, I understood that an easy setup process is going to be of great value as well. What I am specifically talking about is installing OpenCV for Python applications on Google Coral Dev board. Coral was meant to run inference at incredible speeds. But of what use is Coral to me if I can’t even process video frames with OpenCV?

While everything worked flawlessly on a desktop computer, I went to great lengths when installing OpenCV 4.1.1 on Coral. If you are in the same situation, you might have just found a solution! While there is a guide which I followed thoroughly, it didn’t quite work for OpenCV 4.1.1.

Therefore, this post explains how to install OpenCV 4.1.1 on Google Coral Dev board and it is an extension to this and this guide, but with changes and explanations of what and why I changed it.

Requirements

Here is a list of requirements you should do before following this guide:

  • Flash a system image on the board.

  • Connect to your device using either ssh or serial console.

  • Install OpenCV dependencies. Here is how to install everything that is needed for a successful OpenCV build:
    sudo apt-get install build-essential cmake unzip pkg-config
    sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
    sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
    sudo apt-get install libxvidcore-dev libx264-dev
    sudo apt-get install libgtk-3-dev
    sudo apt-get install libatlas-base-dev gfortran
    sudo apt-get install python3-dev
    
  • Install git:
    sudo apt-get update
    sudo apt-get install git
    

Quick solution

  1. Download the prebuilt libraries from this repository.
      git clone https://github.com/pjalusic/opencv4.1.1-for-google-coral.git
    
  2. Copy the cv2.so file to /usr/local/lib/python3.7/dist-packages/.
      cp opencv4.1.1-for-google-coral/cv2.so /usr/local/lib/python3.7/dist-packages/cv2.so 
    
  3. Copy other .so files to /usr/local/lib/.
      sudo cp -r opencv4.1.1-for-google-coral/libraries/. /usr/local/lib 
    
  4. Check if it works.
      python3
      >>> import cv2
      >>> cv2.__version__
      '4.1.1'
    

If it works, you are done. You can now link cv.so file to your virtual environment if you’d like. However, if you prefer building OpenCV yourself or it didn’t work using the quick solution, here is a guide.

Build your own libraries

  1. Create a swap file for the build.
      sudo fallocate -l 2G /swapfile
      sudo chmod 600 /swapfile
      sudo mkswap /swapfile
      sudo swapon /swapfile
    

    Google Coral has only 1GB of RAM which is less than required for the build job. Therefore, we create a swapfile. I am using a 2GB swap file to eliminate any possibility of errors while building, which occurred when using only 1GB swap file.

  2. Mount USB formatted to ext file system (for formatting try this for mac, or this for linux).
      sudo /sbin/fdisk -l
      # at the end is your device, for instance /dev/sda1
      sudo mount /dev/sda1 /mnt
    

    Instead of USB you can use an SD card. Whichever works, but it should have at least 4GB of memory.

  3. Download OpenCV 4.1.1.
      cd /mnt
      wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.1.zip
      wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.1.zip
      unzip opencv.zip
      unzip opencv_contrib.zip
      mv opencv-4.1.1 opencv
      mv opencv_contrib-4.1.1 opencv_contrib
    
  4. Make changes in opencv/cmake/OpenCVFindLibsPerf.cmake file. Before compiling OpenCV, you need to do something extra to avoid errors. This might be only necessary for version 4.1.1, but it may help with others as well. I created a public gist containing a git diff patch file. Clone the repository into your cmake folder and apply changes.
      cd opencv/cmake
      git clone https://gist.github.com/pjalusic/7feb9cd722a437c0f6eea34622ca44c8.git
      cp 7feb9cd722a437c0f6eea34622ca44c8/OpenCVFindLibsPerf.diff .
      git apply OpenCVFindLibsPerf.diff
    

    If there are errors when applying the patch, double check you downloaded OpenCV 4.1.1 or do this by hand. There are exactly 5 lines commented. If it was successful, part of OpenCVFindLibsPerf.cmake file from line 43 should look like this:

      if(WITH_EIGEN AND NOT HAVE_EIGEN)
     find_package(Eigen3 3.0.0)
     if(Eigen3_FOUND)
       #if(TARGET Eigen3::Eigen)
         # Use Eigen3 imported target if possible
         #list(APPEND OPENCV_LINKER_LIBS Eigen3::Eigen)
         #set(HAVE_EIGEN 1)
       else()
         if(DEFINED EIGEN3_INCLUDE_DIRS)
           set(EIGEN_INCLUDE_PATH ${EIGEN3_INCLUDE_DIRS})
           set(HAVE_EIGEN 1)
         elseif(DEFINED EIGEN3_INCLUDE_DIR)
           set(EIGEN_INCLUDE_PATH ${EIGEN3_INCLUDE_DIR})
           set(HAVE_EIGEN 1)
         endif()
       #endif()
     ...
    

    As you can see there are a few changes. The first one is using Eigen3 3.0.0 instead of Eigen QUIET. Secondly, we comment out one if check which gave errors. What all of this does is fixes the compatibility problem with Eigen library because CMake can’t see it. Mind the last comment on endif().

  5. Create a python virtual environment.
      cd ~
      wget https://bootstrap.pypa.io/get-pip.py
      sudo python3 get-pip.py
      sudo pip install virtualenv virtualenvwrapper
      sudo rm -rf ~/get-pip.py ~/.cache/pip
    
  6. Add the following lines to ~/.bashrc file (ie. nano ~/.bashrc) for virtualenv and virtualenvwrapper.
      export WORKON_HOME=$HOME/.virtualenvs
      export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
      source /usr/local/bin/virtualenvwrapper.sh
    
  7. Source that file.
      source ~/.bashrc
    
  8. Create a virtual environment, activate it and install numpy (for some reason I was getting errors until I connected to the device through ssh).
      mkvirtualenv cv -p python3
      workon cv
      pip install numpy
    
  9. Compile OpenCV 4.1.1.
      cd /mnt/opencv
      mkdir build && cd build
      cmake -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_INSTALL_PREFIX=/usr/local \
      -D INSTALL_PYTHON_EXAMPLES=ON \
      -D INSTALL_C_EXAMPLES=OFF \
      -D OPENCV_ENABLE_NONFREE=ON \
      -D OPENCV_EXTRA_MODULES_PATH=/mnt/opencv_contrib/modules \
      -D PYTHON_EXECUTABLE=~/.virtualenvs/cv/bin/python \
      -D ENABLE_FAST_MATH=1 \
      -D ENABLE_NEON=ON -D WITH_LIBV4L=ON \
      -D WITH_V4L=ON \
      -D BUILD_EXAMPLES=ON ..
    
  10. Start building.
      make -j$(nproc)
    

    This will utilize all cores for the build process. It, however, still takes a couple of hours.

  11. Install OpenCV.
      sudo make install
      sudo ldconfig
    
  12. Link to the virtual environment.
      cd /usr/local/lib/python3.7/dist-packages/cv2/python-3.7
      sudo mv cv2.cpython-37m-aarch64-linux-gnu.so cv2.so
      cd ~/.virtualenvs/cv/lib/python3.7/site-packages/
      ln -s /usr/local/lib/python3.7/dist-packages/cv2/python-3.7/cv2.so cv2.so
    
  13. Test it.
      workon cv
      python
      >>> import cv2
      >>> cv2.__version__
      '4.1.1'
    

Well, this was a bit painful and a long process that took too much time. Obviously, the next step is to speed up the build process. How? Well, an idea is to move the build process to an ARM device in the cloud. But that’s a story for another blog post.

Additional information, notes and problems

  • Build OpenCV in background.

    If you would like to do any process in the background, you should check tmux or similar tools. Here is how to do it using tmux:

    • sudo apt-get install tmux
    • tmux → creates a new session. Run workon cv once again. Run make -j$(nproc) command while in the session
    • Control+b, d → leave session
    • tmux ls → show sessions
    • tmux attach -t 0 → return to that session
  • Permission errors while building OpenCV.

    If you encounter permission errors while compiling, building or installing OpenCV, you could always go with the root user option. So before building or opening session with tmux do sudo su. That will also mean that any sessions you create will be bound to root user.

  • Problem with reflashing.

    I stumbled on a serious issue when trying to reflash the device to test my tutorial once again. Following this official guide on reflashing, my Mac was stuck at mdt devices step. Nothing was showing up. Apparently, it is an ongoing issue. My solution was just switching to Windows computer. To follow the guide and successfully reflash the device all I needed to do was:

    • Install python3.
    • Install mdt with pip3.
      pip3 install --user mendel-development-tool
      
    • Get fastboot tool from SDK Platform Tools and manually add fastboot.exe to path

Credits





comments powered by Disqus