Announcement

Collapse
No announcement yet.

Issues with TensorFlow for mineral detection

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Issues with TensorFlow for mineral detection



    I am mentoring a young team this year. All team members with programming experience graduated last year, so this year

  • #2
    I am mentoring a young team this year. All team members with programming experience graduated last year, so this years team is using Blocks since not all current members have knowledge of Java.
    They had been using the Rev color sensor for mineral detection, but they struggled to get better than 80% reliability due to the need to be so close to the mineral to get an accurate read. So when the TensorFlow Lite addition was added last week, they were super-excited as it appeared it would both simplify the detection process as well as offer the possibility of improving the reliability of detection.
    Sadly, though, they only had one week to implement the routines before competition this last weekend, and it didn

    Comment


    • #3


      I am mentoring a young team this year. All team members with programming experience graduated last year, so this years team is using Blocks since not all current members have knowledge of Java.
      They had been using the Rev color sensor for mineral detection, but they struggled to get better than 80% reliability due to the need to be so close to the mineral to get an accurate read. So when the TensorFlow Lite addition was added last week, they were super-excited as it appeared it would both simplify the detection process as well as offer the possibility of improving the reliability of detection.
      Sadly, though, they only had one week to implement the routines before competition this last weekend, and it didnt work very well, only about 50%, so they did poorly in the competition. They are embarking on trying to figure out if it can be improved, but from our experience the sample strategies and code sample are unusable in the current implementation. I thought I would share our findings since the TensorFlow addition to blocks is so new and I am not sure how many teams have experimented or implemented this yet.
      The sample routines are based on a landscape approach to view all three minerals and locate the position of the gold mineral. This presents a challenge to mount the phone, as it must be high up close to the 18 inch limit as well as towards the back of the robot to get all three minerals in view. The phone must be angled properly as not to view the minerals in the crater on the one side of lander.
      Here is a summary of the issues we found in using the sample approach:
      - TensorFlow doesn

      Comment


      • #4


        I am mentoring a young team this year. All team members with programming experience graduated last year, so this years team is using Blocks since not all current members have knowledge of Java.
        They had been using the Rev color sensor for mineral detection, but they struggled to get better than 80% reliability due to the need to be so close to the mineral to get an accurate read. So when the TensorFlow Lite addition was added last week, they were super-excited as it appeared it would both simplify the detection process as well as offer the possibility of improving the reliability of detection.
        Sadly, though, they only had one week to implement the routines before competition this last weekend, and it didn

        Comment


        • #5


          I am mentoring a young team this year. All team members with programming experience graduated last year, so this years team is using Blocks since not all current members have knowledge of Java.
          They had been using the Rev color sensor for mineral detection, but they struggled to get better than 80% reliability due to the need to be so close to the mineral to get an accurate read. So when the TensorFlow Lite addition was added last week, they were super-excited as it appeared it would both simplify the detection process as well as offer the possibility of improving the reliability of detection.
          Sadly, though, they only had one week to implement the routines before competition this last weekend, and it did not work very well, only about 50%, so they did poorly in the competition. They are embarking on trying to figure out if it can be improved, but from our experience the sample strategies and code sample are unusable in the current implementation. I thought I would share our findings since the TensorFlow addition to blocks is so new and I am not sure how many teams have experimented or implemented this yet.
          The sample routines are based on a landscape approach to view all three minerals and locate the position of the gold mineral. This presents a challenge to mount the phone, as it must be high up close to the 18 inch limit as well as towards the back of the robot to get all three minerals in view. The phone must be angled properly as not to view the minerals in the crater on the one side of lander.
          Here is a summary of the issues we found in using the sample approach:
          - TensorFlow does not t work very well, and is very inconsistent. With the phone viewing all three minerals captured in the image perfectly centered, it will randomly fail to detect all three. It seems to have the most problem with the white wiffleballs. If you twist the white ball ever so slightly, then it will pick it up. We have experimented with the confidence factors, to no avail. I have a theory that the image models were not built properly, and different angles of both the wiffleballs and the yellow blocks result in TensorFlow not identifying them as minerals.
          - TensorFlow Lite is new, and is apparently unstable. We have random lockups and freezes of the op mode during the sensing period, not very often, maybe once in 20 trials.
          - Once TensorFlow is initialized, you have a very short period of time before you can start, or the program will freeze. (this is every instance, so this is a major issue) Last Saturday, there were three matches where the start got delayed, which meant we created further delay as our team had to reboot the controller. We likely will move the initialization to after start for this reason, slowing down the op mode.
          We may have to resort to using TensorFlow as an improved color sensor due to the limitations described above. This will be much slower than using a color sensor.
          I would be interested to hear if anyone else has tried to use TensorFlow in Blocks.








          Comment


          • #6
            Sorry for the multiple posts, I was unaware that apostrophes are forbidden, lol

            Comment


            • #7
              I'm not familiar with Blocks, so I don't know if this is possible, but you might be able to modify the program to use only two of the minerals. Assuming that the phone always sees the same two minerals, you can reason out what the third mineral is.

              You might also be able to make the program rely only on the gold mineral, assuming that the phone always sees the same two minerals. If, for example, the phone always sees the left two minerals and the gold mineral is detected on the right side of the screen, then the gold mineral must be the middle. If the gold mineral is not detected, then it must be in the location which the phone cannot see.

              One last suggestion, I noticed that TensorFlow can take some time to detect the minerals. If possible, consider monitoring TensorFlow for a few seconds before moving on.

              Comment


              • #8

                For reliable results from the TensorFlow Example OpMode (in Blocks), make sure the phone is set to Auto-rotate.

                This is clearly explained here:
                https://github.com/ftctechnh/ftc_app...ge-orientation

                The overall OpMode documentation is here:
                https://github.com/ftctechnh/ftc_app...ection-Op-Mode


                Without this, your tests so far have displayed the Gold mineral's relative vertical position in the image, rather than its relative horizontal position. This would randomly seem 'right' or 'wrong', creating much frustration. Been there!

                If for some reason you want the phone locked to Portrait mode, in the Example code (in Blocks) simply change the 3 instances of Recognition.Left to Recognition.Top

                I will post this note in the several threads reporting unreliable TensorFlow results.

                Comment


                • #9
                  Thanks, but this is not the issue. Our phones have been set in auto rotate, as this was one of the first things we had to do in setting this up.

                  Comment


                  • #10
                    I'm sorry that the TensorFlow object detection is inconsistent. The training data was captured on the phones of robots running in a game this past summer. As far as I know, images of whiffle balls and yellow cubes were captured from many different angles.

                    The sample opmode is not meant to be the perfect way to detect the gold mineral, but is to show how you might do it. If your robot's phone can't see all the minerals in one frame, you may have to drive or rotate the robot in order to view the minerals one (or two) at a time. You might also be able to use a webcam instead of the phone's' camera.

                    I haven't experienced the random lockups and freezes that you describe. What phones are you using? If you don't show the monitor screen (set enableCameraMonitoring to false in the TensorFlowObjectDetection.initialize block), does it still lockup/freeze?

                    Comment


                    • #11
                      Thanks for the feedback. We are now using a different approach, only looking for the gold mineral relative X position in the camera frame as it seems pretty reliable. We will give this a shot, and have backup routines that simply look at one mineral at a time much closer to the mineral. If you get a large image of each mineral in the camera, it is 100% reliable. (and much better than using the color sensor!)

                      We are using Moto G 2nd gen phones. I had not thought about disabling the monitor screen, as we had it enabled to see where it was failing. We will give this a shot next week and see if the lockup issue improves.

                      Comment

                      Working...
                      X