Projektarbeiten

At the Robotics Lab at the Fachhochschule Wedel, we like to use ArUCO markers. In particular the function cv::aruco::detectMarkers provided by OpenCV. However, on some systems, we experience performance problems. Since the systems we are working on do have powerful GPUs, I would like to use CUDA. There is a project that allegedly implements ArUCO marker detection with CUDA. However, it does not work for the markers we have and the code looks so messy I cannot really read it: https://github.com/nbergst/cuda_aruco

My preliminary research using gporf leads me to believe that I found a repeated call to cv::adaptiveThreshold. This may offer an opportunity for a speed-up, since about a third of the total time is spent there:
One of the first things you should do is verify my findings by using gporf yourself. For details, look at the parallel_for-loop in _detectInitialCandidates calling _threshold. cv::findContours looks also pretty costly in terms of run-time, but I have no idea on how to speed that up.

I would like to move the repeated call to cv::adaptiveThreshold into the GPU using CUDA. I have no experience with CUDA and do not know how to use it efficiently in this context. I could imagine that one would upload the input image into the VRAM once, then the GPU runs the loop, creating N images (or one image with N layers, or one image of N-times the size) which are then downloaded back to the RAM. The result must be checked for correctness: The traditional implementation and our CUDA implementation should calculate the same images. The result also must be evaluated using gporf to see whether it actually offers an improvement in terms of speed. The implementation can be done in a relatively non-intrusive manner and may be puglished. If you manage to write code so good that it is accepted into the OpenCV project, I will give you the best possible grade. ;)

So far, the result should help us with all our CUDA-enabled systems. For an additional and optional challenge, we can take a closer look at our Nvidia Jetson TX2 system. It features a powerful GPU and a relatively weak CPU. For this reason, the camera (it is called "argus", but I do not know if the name refers to the camera, the interface or a library) writes images directly into the VRAM. This may offer opportunity for a huge increase in performance since we can bypass the (slow) CPU for everything up to (but excluding) the call to cv::findContours. I did some preliminary research on this topic. Find my test programs attached. Since I do not have any experience with CUDA, the problems I ran into are plentiful. I took code and information from these sources:

https://www.ximea.com/support/wiki/apis/Using_OpenCV_with_CUDA_on_the_Jetson_TX2
https://medium.com/@aleozlx/live-streaming-from-jetson-tx2-with-custom-cuda-filters-part-i-ef06366a92c1
https://forums.developer.nvidia.com/t/gstreamer-nvmm-opencv-gpumat/52575/21
https://forums.developer.nvidia.com/t/nano-not-using-gpu-with-gstreamer-python-slow-fps-dropped-frames/107291/22

Unfortunately, because this is the way Nvidia does things, you must use the dated Ubuntu 18 in this context.

All things considered, this project is best done by someone who knows their way around Linux, C++, CUDA, and gporf. Or is really eager to learn about it.

For this project, we need a team that brings combined knowledge of a wide range of fields:

• Networking: The robot is one system, the user is located on a remote system. Movement commands must be sent to the robot. Video must be sent from the camera to the viewer.
• VR Media: The spherical video feed must be displayed somehow.
  I can imagine you could use a game engine like Unity or Unreal for the display and VR integration, but I do not know how well those engines handle live video from an external source.
  I recently learned about the existence of WebVR, which probably handles live video alright, but I do not know how well it actually works.
  The camera itself might be a challenge. So far, I have found no option to actually use it strictly locally (without YouTube, twitch and a smartphone).
  
• VR Controls: Using VR hardware (like the HTC Vive), we might employ their VR controllers instead of a game-pad.
  We have depth-cameras with OpenNI support. Analysing the user's pose or position is an option.
  We have some LEAP motion sensors. Analysing the users hand position or gesture is an option.
  
• Robotics: The robot's high-level system (a laptop) communicates with the robot's low-level system (a microcontroller). This needs someone who can deal with that kind of hardware-related communications.
  While we do have two Eddie robots, only one of them is in a state which can be used reliably. And even that is using a crude implementation I haphazardly threw together: https://vr-tools.fh-wedel.de:3001/hoe/WXEddie For a stupid proof-of-concept, this is okay. Long term we should have something compatible with ROS2. The other robot suffers from hardware failure. I do have replacement parts, but someone needs to do the installation, set-up and verification.
  

The currently existing proof of concept implementation supports moving the robot with the game-pad. Making this work remotely with a simple UDP server/client approach (gamepad on one PC, the robot's high-level system being the other PC) is too easy for a 5 ECTS project. Creation of a package for ROS 2 "humble" using ROS's built-in communication features with clean code a nice documentation which can then be used for future projects would be okay. I would like to see some measurements illustrating the delay between the remote input and actual movement.