Announcement

Collapse
No announcement yet.

Update on USB Disconnect Issues (unable to detect USB modules during scan)

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Update on USB Disconnect Issues (unable to detect USB modules during scan)

    Hi Folks,

    A number of folks have reported issues with the ability for their robot controllers to "see" the Modern Robotics (MR) USB attached modules (Legacy, Core Device Interface, DC motor and servo modules). The folks at Modern Robotics, QualComm and FIRST have been working to try and troubleshoot this problem. Note that not everyone has been encountering this problem, but we recognize that it is a real problem and there are several users who run into this situation regularly.

    Based on the collective testing/experience of the folks at MR, QTI (QualComm Technologies Inc) and FTC we believe that there are some different scenarios that can cause communication (comm) issues between the phone and the USB modules. We do not yet have any definitive answers for these issues, but we do have some ideas on what might be the cause for some of these issues.

    I'd like to give the community an update and provide some recommendations on how to troubleshoot and mitigate these comm errors.

    ************************************************** *******

    1. USB connections - We feel that a fair number of the reported issues of loss of comm to one or more USB device are related to USB connection issues (physical USB connections). Particularly for the case where a team is driving a robot and all of the sudden their system Emergency stops (E stops) and they receive an error from the robot controller that it had trouble communicating with a Modern Robotics device. We suspect that these issues are caused by an intermittent disconnect of the USB connection. Sometimes this occurs when the robot is jolted, but other times this occurs when the robot is just cruising slowly along.

    In this situation the solution is pretty straight forward. We recommend that all of the modules/devices are properly secured to the frame of the robot and the USB cables are secured/strain relieved so they will not vibrate or move during a match. This is especially important for the connection to the Micro USB port on the Android phone. The micro USB port unfortunately isn't the best connector for this application, but it is the connector that we have to work with. Pay attention to the connection between the Power Module and this Micro USB port. Ideally it should click into place when you insert the adapter cable into the Micro USB port. (there are two little tabs that are under tension on the male end of the connector and they are supposed to click into place when the connector is inserted into the port). The cable should be strain relieved so that it will not vibrate or move during a match. If you are experiencing intermittent comm issues when your robot is moving, you should definitely check the connection between the phone and the Power Module, and you should also check all of the other USB connections to make sure they are secure.

    2. Battery Voltage - People have reported this to us, and we have seen this in our pilot testing and demos over the past year, but if the voltage on your main 12V battery drops too low (below 12V) we suspect that this can cause USB module disconnect issues. In my personal experience, the battery has to be fairly low before this seems to occur. The Power Module has a built in voltage regulator that takes the input voltage and steps it down to 5V for the USB modules. According to Modern Robotics, the Power Module can drop down to a fairly low value (at least 9Volts, which would be extremely low for a 12V battery) before the Voltage regulator would not be able to power the USB ports. However, we suspect that if your voltage level on your main battery is low, then there might be situations where a large current draw (for example, when the DC motors switch direction from forward to reverse) could drop the input voltage low enough that the Power Module fails to reliably power the USB ports.

    Note that when we were doing testing for supporting the legacy Matrix 9.6 V motor/servo controller, we tried driving a 12V Motor controller and the Power Module using a 9.6V (2000 mAh) battery. When we did this, the simple square bot (two 9V motors connected to the 12V motor controller) could drive in one direction (forward or backward) but when we tried to reverse direction, the motors would stop and we'd receive an error message indicating a loss of communication with the Modern Robotics device. We suspect that the reversal of the motor caused a drop in the battery voltage that made the Power Module not be able to power the USB bus reliably.

    The solution to situation #2 is also pretty straight forward. Power Management is important. Monitor your 12V battery voltage and don't let it fall too low (around or below 12V). Perhaps having two 12V batteries (and charging the spare when the other is in use) is a good option if your budget permits this. If you are using a Matrix 9.6V battery to power your system, you should only be driving the Matrix 9.6V motor/servo controller. You do not want to drive a 12V motor controller with the 9.6V battery because it can cause comm errors. Also be sure to keep your Matrix 9.6V battery charged and have a spare on hand if budget permits.

    3. This is the situation when a user experiences intermittent and unexplained failures to detect one or more USB modules on the Power Module. It can occur when the user turns on the Power Module and the Robot Controller begins to do its scan but never is able to detect one or more of the USB modules on the Power Module's USB bus.

    This is a situation that can be extremely frustrating to the user. The user will sometimes be able to detect all modules reliably. At other times they cannot detect one or more module reliably. Power cycling the robot, swiping the app close, disconnecting the cable do not seem to help.

    I spoke with both QualComm and Modern Robotics at length this week about this scenario. We are looking at the code to see if there is an issue with the code. However, Modern Robotics indicated that if teams are having this problem, where they are intermittently unable to detect one or more USB module (and power cycling the robot, disconnecting the phone, swiping the app, rebooting the phone all do not work reliably to fix the problem), then the teams that are experiencing this problem should contact Modern Robotics.

    Modern Robotics has been diligently testing various scenarios and configurations related to this problem and they suspect that they have identified a situation where some of the earlier Power Modules (but not all, just a limited number) might have an issue which can cause this type of behavior. I am unable to provide specifics, but the condition is similar to a race condition where the USB chips within the Power Module might be waiting indefinitely to receive a reset signal as they are powered on, causing the situation where one or more of the devices are not able to be detected on the USB bus.

    If a team is having issues with being unable to detect one or more of the USB modules and this behavior is intermittent and does not appear to be related to USB connection issues or low voltage issues, then that team should reach out to the Modern Robotics support team and they can troubleshoot and resolve the problem. It might require that the team send back their Power Module to have a component adjusted/replaced on the device.

    I would like to emphasize that Modern Robotics believes that it is an isolated number of Power Modules that could potentially experience this issue. If you are experiencing USB comm errors, first make sure the issue isn't caused by loose connections or low battery voltage. If you do suspect that there is an issue that you is not caused by a vibrating or loose USB cable or low battery voltage, contact Modern Robotics and their support team.

    ************************************************** *******

    I'd like to thank the member of the community who have been alerting us to the issues they have observed with the new system. I'd like to thank Modern Robotics and QualComm for spending a good amount of time troubleshooting and trying to reproduce these issues.

    As more information becomes available, I'll update the board. We will continue to test and troubleshoot this issue.

    Tom
    Last edited by JoAnn; 10-15-2015, 04:11 PM.

  • #2
    Any information on what range of CPDM serial numbers might be affected?

    Regards,
    Joe Higham
    Mentor, 18 Techno Chix
    Joe Higham
    Mentor, 18 Techno Chix

    Comment


    • #3
      Originally posted by jphigham View Post
      Any information on what range of CPDM serial numbers might be affected?

      Regards,
      Joe Higham
      Mentor, 18 Techno Chix
      Hi Joe,

      That would be a question for Modern Robotics. I'll also ask and see if I can get a serial number range, but feel free to ping MR about this.

      Tom

      Comment


      • #4
        Originally posted by jphigham View Post
        Any information on what range of CPDM serial numbers might be affected?

        Regards,
        Joe Higham
        Mentor, 18 Techno Chix
        Joe - PS I used to live in Croton Falls NY and we'd often drive through Patterson when we would go to hike, etc.. That's a beautiful area of NY!

        Tom

        Comment


        • #5
          Hi Tom,

          Issue #1 seems the most severe since it can happen in the middle of a match. I don't mean to suggest that the other issues can hurt teams esp. if ther are unable to resolve via reboot of the various components... It's just that one solution will not remedy all of these problems.

          So back to Issue 1 -
          If you suspect that this issue can be triggered by even the slightest (temporary) disconnect from the USB connection then you may want to change your E-stop rules to be a little more optimistic in that respect (i.e. attempt at several retries to rescan without blindly shutting down power to motors or crashing the RC App). This may be a viable contingency in absence of a reliable way to secure the USB connections from this "jiggle effect".

          -David

          Comment


          • #6
            Recently, our control system developed a communication problem as stated in this thread. After reading this thread, it looks like we are not an isolated case. At first when we received the system, it just happened once in a while. But now, it is getting worse. In fact, I can't get it to connect reliably at all today. Since we have two phones and both phones started to fail the same way when used as the Robot Controller. That makes me think the fault may be the micro-USB to mini-USB cable. When I charged the phone with a USB-microUSB cable, the insertion still felt loose but not as bad as when plugging the phone to the Core module. It felt so loose that it did not even "click". I have ordered replacement cables, but they won't come for a few days. So I won't know for sure until then. BTW, our control system is not even mounted on a real robot, so there is no issue with the cable getting disconnected while the robot is moving. We have the control system mounted on a piece of polycarb and just playing and learning with programming. Since it is not on a robot, the system hasn't been subjected to all the harsh environment that a running robot may experience. I am starting to worry about the reliability of the fragile micro-USB connector on the phone. I don't think it is designed for the frequent connect/disconnect cycles that we will subject it to for robotics competition and development. At least I felt the ZTE phones are a little too flimsy. Is there a better phone that you can recommend? I have bought a Nexus 7 tablet despite I was warned that it is not compatible with the FTC software because it's on Lollipop not KitKat. However, I succeeded in using it as the Driver Station and having no problem whatsoever. May be the incompatibility is with the Robot Controller software, not the Driver Station software. In any case, I felt the micro-USB connector on the Nexus 7 is a lot more robust (i.e. tighter) than the ZTE phone. Would you recommend using a Nexus 7 on the robot controller side as well? If not, what other devices that are more robust that we can use on the Robot Controller side?
            BTW, the ZTE phone's battery also runs down surprisingly fast comparing to all the other smartphones I have used in the past.

            Comment


            • #7
              Originally posted by mikets View Post
              Recently, our control system developed a communication problem as stated in this thread. After reading this thread, it looks like we are not an isolated case. At first when we received the system, it just happened once in a while. But now, it is getting worse. In fact, I can't get it to connect reliably at all today. Since we have two phones and both phones started to fail the same way when used as the Robot Controller. That makes me think the fault may be the micro-USB to mini-USB cable. When I charged the phone with a USB-microUSB cable, the insertion still felt loose but not as bad as when plugging the phone to the Core module. It felt so loose that it did not even "click". I have ordered replacement cables, but they won't come for a few days. So I won't know for sure until then. BTW, our control system is not even mounted on a real robot, so there is no issue with the cable getting disconnected while the robot is moving. We have the control system mounted on a piece of polycarb and just playing and learning with programming. Since it is not on a robot, the system hasn't been subjected to all the harsh environment that a running robot may experience. I am starting to worry about the reliability of the fragile micro-USB connector on the phone. I don't think it is designed for the frequent connect/disconnect cycles that we will subject it to for robotics competition and development. At least I felt the ZTE phones are a little too flimsy. Is there a better phone that you can recommend? I have bought a Nexus 7 tablet despite I was warned that it is not compatible with the FTC software because it's on Lollipop not KitKat. However, I succeeded in using it as the Driver Station and having no problem whatsoever. May be the incompatibility is with the Robot Controller software, not the Driver Station software. In any case, I felt the micro-USB connector on the Nexus 7 is a lot more robust (i.e. tighter) than the ZTE phone. Would you recommend using a Nexus 7 on the robot controller side as well? If not, what other devices that are more robust that we can use on the Robot Controller side?
              BTW, the ZTE phone's battery also runs down surprisingly fast comparing to all the other smartphones I have used in the past.
              The cables are designed to wear faster than the port itself, so after a considerable amount of disconnects/reconnects you will need to replace the cable in order for a "snug" connection to happen.

              Comment


              • #8
                Originally posted by dmssargent View Post
                The cables are designed to wear faster than the port itself, so after a considerable amount of disconnects/reconnects you will need to replace the cable in order for a "snug" connection to happen.
                Owning the system for a little over 2 months and doing a few dozen connects/disconnects and the connector on the cable is worn? I wouldn't say that's a "considerable amount". I would expect that after at least a year of frequent use. I hope the replacement cables I am getting will do better.

                Comment


                • #9
                  To clarify, one question:

                  we are having problems which sound similar, except that one of our two motor controllers shows up pretty consistently; the other one doesn't. The problem is sometimes resolved by cycling the connector between the power distribution unit and the robot controller phone (cycling meaning disconnect / reconnect). Sometimes we have to power cycle the phone. But we so far have NOT needed to touch the USB cable from the power distribution unit to each of the motor controllers.

                  Would these symptoms be viewed as part of the same family of problems, or as something else? If the latter, any advice will be much appreciated.

                  Martin Haeberli
                  Mentor, FTC 7593 TigerBots

                  Comment


                  • #10
                    Originally posted by Tom Eng View Post
                    Joe - PS I used to live in Croton Falls NY and we'd often drive through Patterson when we would go to hike, etc.. That's a beautiful area of NY!
                    New Hampshire is no slouch in the "beautiful area" category either!
                    Joe Higham
                    Mentor, 18 Techno Chix

                    Comment


                    • #11
                      Originally posted by mhaeberli View Post
                      To clarify, one question:

                      we are having problems which sound similar, except that one of our two motor controllers shows up pretty consistently; the other one doesn't. The problem is sometimes resolved by cycling the connector between the power distribution unit and the robot controller phone (cycling meaning disconnect / reconnect). Sometimes we have to power cycle the phone. But we so far have NOT needed to touch the USB cable from the power distribution unit to each of the motor controllers.

                      Would these symptoms be viewed as part of the same family of problems, or as something else? If the latter, any advice will be much appreciated.

                      Martin Haeberli
                      Mentor, FTC 7593 TigerBots
                      Hi Martin,

                      I would like to make sure I understand the problems you are seeing. Your Robot Controller sometimes has problems seeing one of the two motor controllers. To rectify the problem you disconnect the cable from the Android phone that connects it to the Power distribution module? Does this problem occur when the robot is driving (even if it's driving slowly/not aggressively over a smooth surface) or does this happen when you restart your app or restart your power module and the robot controller fails on its attempt to scan for the configured hardware devices?

                      Tom

                      Comment


                      • #12
                        Originally posted by mikets View Post
                        Owning the system for a little over 2 months and doing a few dozen connects/disconnects and the connector on the cable is worn? I wouldn't say that's a "considerable amount". I would expect that after at least a year of frequent use. I hope the replacement cables I am getting will do better.
                        Hi mikets,

                        In my experience with the early versions connector cable (the one that plugs into the Android phone's micro usb port then runs to the Mini port of the Power Module) that is provided with the Modern Robotics Power Module, I found that their male adapter side of the cable was a little loose and not as snug as the Monoprice USB OTG Adapters that we used in our pilot testing last year. This cable that is included with the Power Module seems to be a little looser out of the box than the Monoprice connector in my experience. Hopefully the Monoprice one will last longer for you (they have for our testing here at FIRST).

                        As dmssargent pointed out the cables are designed to wear faster than the actual port so over time, the connectors do seem to get a little more loose. I personally like the Monoprice Micro USB OTG adapter cables because in my experience they tend to have a more positive and snug connection to the port. However, even these cables loose some of their snugness over time (although it does take a fair number of insertion and removal cycles to wear these down in our testing...).

                        Tom

                        Comment


                        • #13
                          Hi Tom,
                          I just got off the phone with Modern Robotics to see about getting our CPDM repaired, and it was mentioned that the OTG cable is unusual in having the master-slave relationship swapped. Can you post a part number for the Monoprice cable so we can be sure that we get the right item?

                          Roy

                          Comment


                          • #14
                            Originally posted by mikets View Post
                            Recently, our control system developed a communication problem as stated in this thread. After reading this thread, it looks like we are not an isolated case. At first when we received the system, it just happened once in a while. But now, it is getting worse. In fact, I can't get it to connect reliably at all today. Since we have two phones and both phones started to fail the same way when used as the Robot Controller. That makes me think the fault may be the micro-USB to mini-USB cable. When I charged the phone with a USB-microUSB cable, the insertion still felt loose but not as bad as when plugging the phone to the Core module. It felt so loose that it did not even "click". I have ordered replacement cables, but they won't come for a few days. So I won't know for sure until then. BTW, our control system is not even mounted on a real robot, so there is no issue with the cable getting disconnected while the robot is moving. We have the control system mounted on a piece of polycarb and just playing and learning with programming. Since it is not on a robot, the system hasn't been subjected to all the harsh environment that a running robot may experience. I am starting to worry about the reliability of the fragile micro-USB connector on the phone. I don't think it is designed for the frequent connect/disconnect cycles that we will subject it to for robotics competition and development. At least I felt the ZTE phones are a little too flimsy. Is there a better phone that you can recommend? I have bought a Nexus 7 tablet despite I was warned that it is not compatible with the FTC software because it's on Lollipop not KitKat. However, I succeeded in using it as the Driver Station and having no problem whatsoever. May be the incompatibility is with the Robot Controller software, not the Driver Station software. In any case, I felt the micro-USB connector on the Nexus 7 is a lot more robust (i.e. tighter) than the ZTE phone. Would you recommend using a Nexus 7 on the robot controller side as well? If not, what other devices that are more robust that we can use on the Robot Controller side?
                            BTW, the ZTE phone's battery also runs down surprisingly fast comparing to all the other smartphones I have used in the past.
                            mikets

                            The Nexus 7's are a nice tablet. You'll need to check on the Q&A forum for an official answer, but I believe that for this season, the recommended phones are the ZTE Speed and the Motorola Moto G. I'm not sure what the ruling is on allowing a different device as a driver station or robot controller is, and I recommend that you post a question on the appropriate Q&A forum.

                            The Nexus 7's do work well, provided that you have the proper version of the Android kernel. Note that there was a bug in Android Lollipop (that will be fixed in Marshmallow, and that is fixed for some, but not all, Android devices in Lollipop) that prevents a Lollipop device for detecting more than one USB module at a time. If your Android device has this bug, it won't be able to see more than one USB module at a time including the Modern Robotics devices and the gamepads. The ZTE Speeds are Kit Kat devices and are unaffected by the bug. The motorola Moto G's have a fix (if you have the latest lollipop updates from Motorola) for their Lollipop phones so they should work, even though they use Kit Kat. The nice thing about the Nexus devices is that Google makes available the older versions of the kernels so it is possible to downgrade a Nexus if the older kernel is still available from Google.

                            Also, regarding the battery life of the ZTE speeds. The ZTE Speeds are doing a lot of work when they are connected to a robot. The Robot Controller is acting as the Wi-Fi Direct Group owner (like virtual access point) and there is a lot of communication going on between the robot controller and the hardware modules and the robot controller and the driver station. Also, the driver station is powering the two gamepads and the processors on both devices are active.
                            Last edited by Tom Eng; 10-15-2015, 10:51 AM.

                            Comment


                            • #15
                              Originally posted by Tom Eng View Post
                              The Nexus 7's do work well, provided that you have the proper version of the Android kernel. Note that there was a bug in Android Lollipop (that will be fixed in Marshmallow, and that is fixed for some, but not all, Android devices in Lollipop) that prevents a Lollipop device for detecting more than one USB module at a time. If your Android device has this bug, it won't be able to see more than one USB module at a time including the Modern Robotics devices and the gamepads. The ZTE Speeds are Kit Kat devices and are unaffected by the bug. The motorola Moto G's have a fix (if you have the latest lollipop updates from Motorola) for their Lollipop phones so they should work, even though they use Kit Kat. The nice thing about the Nexus devices is that Google makes available the older versions of the kernels so it is possible to downgrade a Nexus if the older kernel is still available from Google.
                              If this is a 2013 Nexus 7, Marshmallow is now out for them, and the OTA has been rolling out this week. (That said, for competition purposes, if the Nexus 7 is allowed, the rules require sticking to Kit Kat.)

                              Comment

                              Working...
                              X