Announcement

Collapse
No announcement yet.

Retraining TensorFlow with Videos of Minerals on Red and Blue Tape?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Retraining TensorFlow with Videos of Minerals on Red and Blue Tape?

    Comrade 17 Tom Eng A few teams have been reporting that TensorFlow is failing when the minerals are on the red and blue tape in the sampling field at the start of a match (e.g., https://new.reddit.com/r/FTC/comment...orflow_issues/). The google github repo describing the TensorFlow training for FTC minerals (https://github.com/google/ftc-object-detection) contains training and validation videos that show minerals on bare gray tiles and not on the red and blue tape. Given this, it seems likely that the learned models are for minerals on gray tiles and as a result that the models may be struggling to detect minerals on the red and blue tape in the sampling fields.

    Would it be possible for the FTC Tech team to retrain the TensorFlow models using videos of gold and silver minerals on both red and blue tape as they appear in the sampling field at the start of matches, and release an updated model so that TensorFlow can have improved performance for mineral sample field detection? Ideally, training would use videos of minerals in different orientations on the different tape colors and in differing lighting conditions to help improve the robustness of the detection.

    Thanks for your consideration.
    CHEER4FTC website and facebook online FTC resources.
    Providing support for FTC Teams in the Charlottesville, VA area and beyond.

  • #2
    Perhaps teams should have to put in some work instead of being given the entire challenge solution - or at least a significant portion. The TensorFlow examples pretty much show how to identify the gold mineral - leaving only determination of relative location in the image. This already seems like a mid-season leveling of the playing field against teams that have developed consistent image processing solutions. No need to do all the work for them.

    Comment


    • #3
      Our team has done just that, we have done a ton of work to get it to work reliably. On one hand, I agree somewhat with the point that FTC7253 has made, because our hard work might differentiate us from our competitors. On the other hand, putting out a sample routine that does not really work very well does not help those that could learn from others on how to improve their robot development. At the very least, the sample code tutorial should state that the example code is not very not reliable, and teams will need to use other tricks to get the code to work reliably.

      Comment


      • #4
        I don't have an answer about whether FTC can retrain the models, but I wanted to let you know that I saw this. I didn't want you to think I missed it or was ignoring it.

        Comment


        • #5
          Hi Folks,

          I did some additional testing in response to this post. I used 2" gaffer squares on an actual field and used the default Blocks TensorFlow sample op mode to identify and track the objects. I did not notice any issue with using the TensorFlow detector to accurately identify the mineral targets (silver or gold), in spite of the presence of the gaffer tape squares. I tried with the minerals placed on the center of the squares (so they mostly covered the squares) and I tried with the minerals only partially on the squares. The lighting was normal (not super bright and not super dark) and the TensorFlow detector had no issues identifying the minerals. The bounding boxes were accurate and the confidence levels were very high.

          Is it possible that there are other issues (such as having additional game elements visible in the image) which might be causing the detection issues? Would you be able to provide me with a test scenario that I can attempt to reproduce where the TensorFlow detector is confused by the red or blue gaffer tape squares?

          Tom
          Last edited by Tom Eng; 11-27-2018, 04:07 PM.

          Comment


          • #6
            Some additional information - I did some more quick tests with the minerals completely off the squares. In this case, I did start to see false positives where the TensorFlow target detector identified the squares as possible minerals. However, this only occurred if the minerals were mostly off the squares. If the minerals were only partially off center, i did not notice false positives. Is the issue that teams have trouble tracking the gold mineral once it's been moved off of the square?

            Comment


            • #7
              I think low lighting that many of us work in might contribute? Also if the square tape is recognized as a block before the balls are placed, it seems persistent. Is there a way to increase the confidence interval or such?

              Originally posted by Tom Eng View Post
              Some additional information - I did some more quick tests with the minerals completely off the squares. In this case, I did start to see false positives where the TensorFlow target detector identified the squares as possible minerals. However, this only occurred if the minerals were mostly off the squares. If the minerals were only partially off center, i did not notice false positives. Is the issue that teams have trouble tracking the gold mineral once it's been moved off of the square?

              Comment


              • #8
                Originally posted by korimako View Post
                I think low lighting that many of us work in might contribute? Also if the square tape is recognized as a block before the balls are placed, it seems persistent. Is there a way to increase the confidence interval or such?
                Hi korimako,

                Yes, low lighting might have a negative effect on the ability of the system to detect the objects. I ran my most recent tests in a normally lit room (fluorescent overhead lights).

                You can specify the minimum confidence threshold when you initialize the TensorFlow library. By default the minimum confidence value is 40% but you can increase this threshold to be more selective when identifying targets. If you set the minimum value too high, however, you might miss targets.

                Also, I have found that for this season's game, if you orient your phone in a landscape (rather than portrait) mode the field of view is wider and the camera can "see" all of the elements in a single frame.

                I hope this helps.

                Tom

                Comment


                • #9
                  Originally posted by Tom Eng View Post
                  Hi Folks,

                  I did some additional testing in response to this post. I used 2" gaffer squares on an actual field and used the default Blocks TensorFlow sample op mode to identify and track the objects. I did not notice any issue with using the TensorFlow detector to accurately identify the mineral targets (silver or gold), in spite of the presence of the gaffer tape squares. I tried with the minerals placed on the center of the squares (so they mostly covered the squares) and I tried with the minerals only partially on the squares. The lighting was normal (not super bright and not super dark) and the TensorFlow detector had no issues identifying the minerals. The bounding boxes were accurate and the confidence levels were very high.

                  Is it possible that there are other issues (such as having additional game elements visible in the image) which might be causing the detection issues? Would you be able to provide me with a test scenario that I can attempt to reproduce where the TensorFlow detector is confused by the red or blue gaffer tape squares?

                  Tom
                  Thanks for checking into this Tom. Here is more info about our setup and what we have experienced. We are using the back camera on our G 2nd gen phones. The phone is mounted on the robot with the camera lens facing downward about 45 degrees or so, and is 10 and a half inches from the face of the lander, 3 and a half inches off center to the right, and 14 inches off the ground.

                  We are sensing all three minerals at once. We lower from the lander, then sense the minerals, with the phone at the above dimensions. On the crater side, we need to have the minerals in the very upper part of the frame, because we can see the minerals in the crater if it is any higher. In our robotics classroom, the lighting is fluorescent lights, and on the standard gray mats with the red or blue squares, it was pretty reliable, because we didn't have the correct crater in place, and had the minerals more centered horizontally (rarely, it would fail to see the left most silver mineral)

                  In our last competition, in the school lunchroom where a practice field was setup, we realized that our camera was looking too high and saw minerals in the crater, and tilted it down so the minerals are at the very upper part of the frame. We started seeing instances where it would not recognize the silver minerals, maybe 20 percent of the time. Then, to our horror, when we moved to the school gymnasium for competition, we failed about 50 percent of the time to see the white minerals, because the gym had high intensity lighting over the basketball court. The light color was much different as well. Maybe mercury vapor?

                  In subsequent testing, we found that several factors contribute to not correctly seeing minerals, having the mineral in a corner of a frame, viewing the mineral at an oblique angle, that is too low, and high intensity light with the mercury vapor lights. It our competition above, it was always the white minerals that failed.

                  We have changed our detection to focus only on gold, and view them closer and higher to detect it. We are batting 100 percent at the moment. We will move our field to the school gym before the next competition to make sure we accommodate changing light levels.

                  Comment


                  • #10
                    11343_Mentor - Thanks for the additional information. Focusing on identifying the Gold Mineral sounds like a good strategy based on your initial experience with varying light conditions. While the demo op mode shows how to determine the relative location of the Gold Mineral when all three minerals are identified, for the autonomous task it is sufficient to identify (then navigate to) the Gold Mineral.

                    Comment


                    • #11
                      Yes, and in addition to it identifying the gold object, it is really cool that it returns the X coordinate, which we are using to locate it within the frame from an absolute sense,

                      Comment


                      • #12
                        We have tested the sample TensorFlow opmode, but have decided to stick with our GRIP-developed openCV image processing pipeline. However, there are some lessons we have learned that can apply to either case.
                        1. We mount our phone low on the bot in landscape with a mount that provides consistent placement/orientation of the camera. This consistent placement is important to help with #2. The low placement also allows using the crater rim as a backdrop to improve sample field mineral contrast, and to block the other minerals that are inside the crater (except for small slithers that can peak thru the gaps).
                        2. We crop the image - throwing away everything that is above the typical sampling field mineral height. We have a couple of additional cropping options from there for the horizontal. We also down sample the image - the resolution from the phone is overkill for this application. Cropping helps remove false positives from the scene - in one competition, when we didn't crop enough vertically for a depot-side run, we found a yellow-shirted judge in the background - with perspective, it matched the color, size, and aspect ration filters. . Cropping and down sampling greatly improve processing time.
                        3. With our low, landscape, flush (90 deg) mount, the camera points straight forward. With the location on our bot (~4 in forward of center) and the location on the bot after landing, we are unable to get all 3 images in the scene at once. Both the Samsung S5 and the Moto G4 have just over 60 deg FOVs. The math just barely doesn't work out to get all 3. Therefore, we have options to either scan for (and crop to) a single mineral at a time, or to turn to be centered on two minerals and scan for the relative location of the gold.
                        4. If using openCV (and in general), try to reduce memory allocation - especially for large chunks like images mats. Instead of a bunch of "new" mats for each processing step/frame, allocate once, and then reuse those chunks. This improves performance and reduces garbage collection hits.

                        So - takeaways: consistently landscape mount phone, crop/down sample image, orient phone to allow blocking other minerals/objects, use algorithm that doesn't require all 3 minerals in scene.

                        Comment


                        • #13
                          Hello,
                          We are a rookie team and have gotten TensorFlow lite integrated into our code. Surprisingly, it was not as difficult as we thought it would be. We'd like to down-sample images like you've suggested but can't find any documentation on how that is done. Any pointers to code samples would be greatly appreciated!



                          Comment


                          • #14
                            Originally posted by tfb View Post
                            Hello,
                            We are a rookie team and have gotten TensorFlow lite integrated into our code. Surprisingly, it was not as difficult as we thought it would be. We'd like to down-sample images like you've suggested but can't find any documentation on how that is done. Any pointers to code samples would be greatly appreciated!


                            TensorFlow and Vuforia both need a high-resolution image to work well. Downsampling will reduce detection performance because they are both based on feature-detection.

                            Comment

                            Working...
                            X