Announcement

Collapse
No announcement yet.

Using deep learning for real-time object detection in FTC

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Using deep learning for real-time object detection in FTC

    Hi FTCers!

    Since FTC has switched to using phones as on-board processors, the ceiling for programming complexity has risen drastically. There are many new and exciting things now available that were impossible with old hardware. One field that benefits greatly from the change is computer vision (CV), which FIRST began egging teams on to use last season with Vuforia. A handful of teams used Vuforia, as well as an amalgamation of other open-source CV platforms like OpenCV. As experience in the community grows, I'm sure more teams will feel comfortable enough to add it to their toolboxes.

    This past summer I spent a great deal of time implementing a neural network on an Android device which could detect game elements using the camera in real-time. The results were promising and I'd like to share them here! Deep learning is a term used to describe a class of algorithms which use structures called neural nets. Over time and with carefully prepared training data, they learn to infer the correct output based on a given input. They (to my knowledge) have never been used in FTC before for image processing, so I'm excited to show teams!

    SO WHAT? Neural nets can detect complex game elements like wiffle balls, yoga balls, and even robots (see attached pictures). Plus, it's incredibly interesting and fun to learn.

    LEARN MORE: There is a lengthy explanation of the project with examples, installation details, and code on my GitHub at https://github.com/kerrj/yoloparser. It is a must-see if you're at all interested in this project. The project's website is http://boxesaroundrobots.com which you can use to develop your own object detector if you wish (this is explained in detail on GitHub).

    QUICKSTART: If you'd like to see a demo in action immediately, download the app on the Play store titled "FTC Neural Net Demo," https://play.google.com/store/apps/d...stin.ftcnndemo. It is capable of detecting red and blue wiffle balls from Velocity Vortex quite robustly, as well as robots with a little less consistency. If you find any bugs please report them by emailing!

    Thanks for reading and I hope I've piqued your interest!
    I don't pretend to be an expert on machine learning/neural networks, but I'll do my best to answer any questions you have!
    ~Justin

  • #2
    I mean, it's interesting, but I'm getting 1300ms recognition time with a Snapdragon S4 Pro, and 700ms with a Snapdragon 800..... So unless new phones with 835 chips in them are going to be allowed, I'm not sure this is a viable option.

    Comment


    • #3
      Originally posted by 4634 Programmer View Post
      I mean, it's interesting, but I'm getting 1300ms recognition time with a Snapdragon S4 Pro, and 700ms with a Snapdragon 800..... So unless new phones with 835 chips in them are going to be allowed, I'm not sure this is a viable option.
      Google will be releasing TensorFlow Lite in an upcoming version of Android Oreo which will be a platform taking advantage of phones' GPUs for computing neural networks instead of relying on the CPU, which has the potential to speed up detection time ~10x. Google and Qualcomm did a demo not long ago showing how using a phone's GPU could speed up large-scale image processing neural nets from .5 FPS to roughly 10 FPS. I certainly agree right now it's rather sluggish, but within the next 6 months I think we could see framerates as high as 10 with Tensorflow Lite. See the links below for more info, and especially watch the embedded video about the Qualcomm. https://www.qualcomm.com/news/onq/20...exagon-682-dsp. https://medium.com/@jrodthoughts/ten...s-64f282cc4819.

      Comment


      • #4
        The deep learning AI you have created is just what I imagined on how to play this year's FTC game. But I have no idea how to use it... after downloading your code and visiting your website I could not find a clear explanation on how to use/implement this into an android phone and get it to work. Could you create something to be able to explain how to get started on using your code?

        --
        Sincerely,
        Adrian Chan
        Team 12828 (Critical Overload)

        Comment


        • #5
          Originally posted by Adrian Chan View Post
          The deep learning AI you have created is just what I imagined on how to play this year's FTC game. But I have no idea how to use it... after downloading your code and visiting your website I could not find a clear explanation on how to use/implement this into an android phone and get it to work. Could you create something to be able to explain how to get started on using your code?
          There's pretty extensive documentation on the github readme, it should be enough to get started. There's also full source code for the app on github as well, let me know if you still can't get it working. The website is mainly for practical use of annotating data and doesn't have documentation, read the readme instead.

          Comment

          Working...
          X