Announcement

Collapse
No announcement yet.

Time to re-think ESD (electro-static discharge) strategy

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Time to re-think ESD (electro-static discharge) strategy

    I don't fully understand how FTC settled on the current ESD rules and field elements, but I can well imagine it was a well intention-ed attempt at minimizing robot-to-robot zaps. The policy is one of isolation: keep all robots isolated from each other as much as possible. Industry does the opposite: tries to keep everything at the same potential to prevent build-up in the first place. Let compare the two methods:

    Grounding Method:
    -> All objects try to ground themselves using a high impedance to a common ground
    -> If possible, increase the air conductivity by keeping humidity up
    -> Use air ionizers if possible to help discharge surfaces that aren't grounded

    Isolation Method:
    -> No one is allowed to ground themselves
    -> All conductive materials need to be protected/kept away from touching foreign conductive materials
    -> Very careful selection of materials in your robot needs to happen in order to minimize charge creation

    Consequences:
    -> All metallic objects (e.g. Robots) become big storage vessels of energy.
    -> Energy builds up and up until the rate of discharge through the air (thanks to moisture) is able to keep it from building up further
    -> The mats are non-conductive and thus all friction with them easily creates energy, making the build-up of energy very easy
    -> Anything that is not well insulated (other robots, beacons, etc...) are now vulnerable to damage
    -> Electronics are not built to handle this massive amounts of energy, thus if they are zapped (e.g. beacons) they will have damage. Most micro-chips are built to handle <= 2kV zaps (human-body model), which is a level you don't feel. The voltage levels you feel are >= 10kV!

    Trying to solve ESD issues becomes very difficult because it is largely dependent on how much moisture is in the air. People in Florida may have little or no issues ever, while people up north in the cold winters will experience lots of issues especially during cold-spells.

    One of the mis-conceptions people may have is that the act of grounding the robots will result in robot-to-robot ESD events. Grounding everything does NOT mean creating low-impedance paths.. quite the opposite, ESD grounding involves very high-impedance conductive surfaces (e.g. 1 Meg Ohms). The purpose of grounding is trickle-off built up charge, not to rapidly discharge it.

    I propose FTC embrace the following policy changes. As FTC grows year to year, these will get harder and harder to implement and thus the sooner FTC bites the bullet the better:
    1) Change the mats to ESD compliant ones! These mats are the key instigator of easy ESD build up. ESD compliant ones will help discharge build-up and help prevent build-up in the first place.
    2) Allow robots to ground themselves to the mat. An ESD grounding strap that is high impedance (and made of soft fabrics) will neither cause damage to the field, nor create low impedance paths for robots.

    FTC has proposed spraying the field with water and/or using an ESD spray on the field. While these quick-fixes are helpful, they change the field's characteristics (friction) and while help, do not prevent the problems as effectively as adopting new mats and policy changes.

    Greg

  • #2
    Originally posted by gszczesz View Post
    I propose FTC embrace the following policy changes. As FTC grows year to year, these will get harder and harder to implement and thus the sooner FTC bites the bullet the better:
    1) Change the mats to ESD compliant ones! These mats are the key instigator of easy ESD build up. ESD compliant ones will help discharge build-up and help prevent build-up in the first place.
    2) Allow robots to ground themselves to the mat. An ESD grounding strap that is high impedance (and made of soft fabrics) will neither cause damage to the field, nor create low impedance paths for robots.
    Couldn't agree more with those suggestions. Spot on. One addition would be ensuring the mats are appropriately earthed either via wired connection (compiant to regulations of course) or ensuring tiles are placed on a charge dissippating surface (i.e. not carpet!)

    Comment


    • #3
      Anti static sprays - their proper use in FTC

      Originally posted by FTC5110 View Post
      Couldn't agree more with those suggestions. Spot on. One addition would be ensuring the mats are appropriately earthed either via wired connection (compiant to regulations of course) or ensuring tiles are placed on a charge dissippating surface (i.e. not carpet!)
      I agree the ideal solution to this problem is to replace the FTC mats. However, conductive mats are expensive.

      At the very least anti-static spray needs to be used on all competition and practice fields. We just came back from East Super Regionals. The organization and volunteers running the event are top notch. We very much appreciate their hard work and dedication.

      Unfortunately, however, the decision was made collectively by ALL regional directors to NOT use anti static spray on the fields. East Supers did spray water every now and then but not according to the recommendations from FIRST, ie: dampen the field and re-apply every 10 minutes.

      The protective mat over the gym floor, coupled with the FTC mats, created an environment rich for ESD events. It was heartbreaking to see so many dead bots in both divisions. Talking to teams, and our own experience was that bots ran perfectly fine on the practice fields where there was no underlay. Go out to the tournament fields and robots experience strange, not repeatable errors when contact was made with other robots, the metal ramp or the field perimeter. End result is a dead robot.

      The recommendation from FIRST on how to use the anti-static spray is not correct. I spoke with the manufacturer of the Tech-Spray Anti Stat 1726QT. Tech-Spray recommends applying their product at the most ONCE every 30 days for DAILY USE in an industrial environment. FIRST says you have to use it before every competition which is NOT recommended. That causes a build up that effects the tile surface. Eventually the mats will need to be washed or replaced as they become too tacky.

      We saw many discouraged teams and kids at East Supers because of the ESD issues. Even the number one seeded alliance in the Tesla division got zapped by this during the semi-finals. This should not be happening. These teams worked incredible hard to make it that far in the competition. We've not seen anything like the number of dead bots in other competitions and these teams would not have made it so far if they had not done everything reasonably possible to build and program robust robots.

      I urge FTC to revisit this issue. Get the manufacturer of these highly engineered anti-static sprays involved in the solution. They most likely will help set firm guidelines for everyone's benefit.

      Comment


      • #4
        I would have to strongly agree. At our local competitions we have not seen a lot of ESD issues because the FTA's have either treated the fields before the event with Staticide or are making sure to water down the field around every 10 minutes or so. The only time we have seen issues is when we neglected to wipe our own robot down before matches.

        That being said we were just informed from the North Super Regional that they will not be using any type of ESD prevention at all. This is very frustrating as a coach to see teams struggle and see all of their hard work dead on the field when the solution is put out there for all tournament directors. In the tournament director manual it even states that the best solution is to use an anti static spray and then to use water if anti static spray is not an option. To a point I do understand the ruling from FIRST in the official forums that they are leaving the decision to use ESD prevention measures up to the tournament directors since some areas are more prone to ESD than others. However, it is very frustrating when our local and state tournaments are taking more care to prevent ESD than our Super Regionals.

        I hope in the future they do address this and try to set some sort of standard that all tournaments should be following to help decrease ESD events.

        Comment


        • #5
          Something does need to be done about the ESD issue.

          I completely agree with this. With the number of hours that goes into this endeavor and to have an ESD issue destroy it is criminal. I can't understand why FTC continues to ignore this problem. My mats measure -6000 volts all the time. I have used commercial anti static spray and it works amazingly well. I think that the least expensive solution would be to conduct tests with a spray and then declare it as the official spray to be used at all tournaments. One bottle of spray cost $20, and in bulk could be less. I'm sure that all coaches and mentors would gladly pay an extra 1$ in entry fees to have the fields sprayed. If teams knew that a certain spray was going to be used all of the time, then they could test with it on their home fields to make sure that the possible change in field friction was designed into their robot. I'd be willing to conduct the tests myself. Living in Colorado makes it easy to find low humidity days to test solutions to this problem. I encourage other coaches and mentors to speak up on this issue. We (teams under the FIRST umbrella) should have some say on this matter.

          Frank Bernett (Mentor for Up-A-Creek team #11260)

          Comment


          • #6
            There was mention above about the effect of the protective covering between the floor and the mats in the competition fields. The layer is typically not present on the practice fields. Does anyone have an idea about how much worse this makes the problem? If someone has the tools to measure the mat voltage, would they be willing to check this with and without the additional layer (tarp or such)?

            Comment


            • #7
              Originally posted by FTC7253 View Post
              There was mention above about the effect of the protective covering between the floor and the mats in the competition fields. The layer is typically not present on the practice fields. Does anyone have an idea about how much worse this makes the problem? If someone has the tools to measure the mat voltage, would they be willing to check this with and without the additional layer (tarp or such)?
              Given the foam mats are ideal insulators, what happens directly below the mats shouldn't affect anything.
              On the other hand, the floor around the field acts as a grounding point for people and the moisture in the air, thus if it is a conductive surface it helps prevent buildup while carrying the robot on/off the field. Consider if the field was on carpet, then every time a person walks to the field and touches the metal wall they'd be charging the wall up, and the robots would have no way to equalize to that charge level except when they touch it.

              Comment


              • #8
                Originally posted by FB_BF View Post
                I encourage other coaches and mentors to speak up on this issue. We (teams under the FIRST umbrella) should have some say on this matter.
                I wholeheartedly agree with idea of standardizing on anti-static spray, as a bare minimum.

                At our state tournament, the fields were not sprayed and the results were disastrous for my team and several others. Our robot died in 4 out of 5 qualifier matches with the dreaded "OpMode stuck in loop()" error. This error had not appeared in the four competitions we played and many hours of practice leading up to the State Championship. It caught us completely off guard, despite having treated our own robot with anti-stat spray and insulating exposed metal. It was the most frustrating experience in my five years mentoring. Our team worked harder and smarter than ever before, performed exceptionally at every meet and tournament to date, and had high hopes to advance to Supers. Those hopes were dashed in their first match when the robot didn't get 10 seconds into TeleOp before locking up completely.

                Many other robots battled the problem throughout the day. In 37 matches I watched, 20 of them had at least one dead robot by the end. I can't say they were all from ESD, but I'm betting the majority were. The FTAs told me they were getting shocked frequently when they touched a robot.

                My other thought on this is that rule on grounding straps should be flipped. If they really make a difference, they should be strongly encouraged. Teams that don't ground should be warned that they run ungrounded at their own risk.

                This problem must be solved by FTC. To ask the kids to work on ESD prevention, which is nearly impossible to replicate reliably, is asking too much. As I said, we didn't see it until State this year. The kids should be thinking about their approach to the game, designing and building, and everything else FIRST encourages - not figuring out all the "tricks" you need to do to prevent shocks. ESD needs to be out of this equation. That being said, we've left our robot intact for the post-season, continue to try to replicate the problems, and devise and test static mitigation strategies. Not having a lot of luck on the replication part though. I'm currently in the market for a static gun I heard about somewhere.

                Chuck Spohr
                5119 Baryons Mentor

                Comment


                • #9
                  Hi Chuck - I am so very sorry 5119 had such an experience. Here are a handful of thoughts:

                  Re: grounding straps. Unfortunately, grounding straps will not work. The physics is counter intuitive, but the mechanism by which a robot charges via ESD is not the reversible via a ground strap. The competition tiles are highly resistive. They will not easily accept current charge from the bot. The mechanism by which they charge is electro-mechanical. So while they seem attractive, grounding straps don't work and only pose a entanglement hazard.

                  Re: static gun. There are such things and they can be rented or purchased for a hefty price. They can also be built - but you would still need the power supply. If I was at work, I could send you the test equivalent circuit for a standard discharge. I believe first did a fair amount of testing with MR hardware. If my memory serves me correct, Thomas Eng spoke about a 6% failure rate (I need to look for the old post). I do not know what the test conditions were nor the hardware under test (E.g.: phones, etc).

                  Re: OpMode Stuck In Loop Error. Please check your phone's log files to see if there is any anomalous things going on. There are reasons why the phone would exit (e.g.: infinite loop) that have nothing to do with ESD.

                  Again, I think that though this problem can't be eliminated (except at huge cost), it can however be substantially mitigated by the use of anti-static sprays - - used according to manufacturers directions. FIRST had it right last year, but went too far with over application. It gunked up the fields and now they've swung too far in the wrong direction - not using at Supers. The sprays were used correctly at WVA States and I can't remember a single electrically dead robot. So, in my mind, the logical place to go is for FIRST to highly recommend the proper use of these sprays.

                  And while I am on a roll - I believe water to be a very poor substitution for anti-static spray.
                  - Water is highly variable in its conductivity. Distilled water is very non-conductive. Add some ions and it becomes semi-conductive. So right off the bat, you don't know what you've got.
                  - Water has high surface tension. It likes itself better than it likes the mat and is more interested in making quasi-non-deterministic semi-conductive islands than it is in making a true conductive film on the mat.
                  - When the water evaporates - it's done. When it is partially evaporated, it's also done, since any conductive surface has now been broken up as the water evaporates unevenly.

                  Sorry to be long winded - but I am equally displeased with the performance of this hardware in a high ESD environment. However, since the hardware is what it is and needs to be affordable, the only thing left to do is to is change the environment.

                  Comment


                  • #10
                    Thank you for your thoughts, FTC4318. I suspected grounding straps might be a bust, but if I can replicate the situation in a controlled environment, I might test it out anyway.

                    Regarding the logs, I did collect them from the phone after the tournament, we did see three motor controllers timeout simultaneously. I'll paste the relevant bit below.. I'm not sure how that could happen besides ESD or loose wiring (we had good, strain-relieved connections and new USB cables with well-fitting connections). Also, the fact that I haven't been able to reproduce this since the weather got warmer and I haven't received any shocks during testing...

                    I know that there's a newer SDK to help with the timeouts, but we hadn't installed it before State, since you generally don't want to fix something that ain't broke. We'd been stable up to that point. Again, if i can reproduce, we'll test this as well. Waiting for another cold snap here in Missouri so I can get some good dry air in the house.

                    It's also possible we have something flaky in our PDM, being the common component to the three simultaneous timeouts. It's a complete mystery to me so far, and, admittedly, static is the easy explanation. When we run this in a humidified environment (my home or at school), we don't see the OpMode stuck in Loop(). Perhaps something in our setup is more susceptible to ESD problems. I just don't know what it is.

                    Here's the log from our last match of the day:

                    03-04 14:24:19.320 2557 3245 W RobotCore: Modern Robotics USB header sync bytes are incorrect
                    03-04 14:24:19.320 2557 3245 W RobotCore: [55 aa 80 40 1e] -> [00 00 00 00 00]
                    03-04 14:24:19.320 2557 3245 W RobotCore: could not read Modern Robotics USB DC Motor Controller [AL00VF7C]: comm timeout
                    03-04 14:24:19.360 2557 2974 W RobotCore: Modern Robotics USB header sync bytes are incorrect
                    03-04 14:24:19.360 2557 3000 W RobotCore: Modern Robotics USB header sync bytes are incorrect
                    03-04 14:24:19.360 2557 2974 W RobotCore: [55 aa 80 40 1e] -> [00 00 00 00 00]
                    03-04 14:24:19.360 2557 2974 W RobotCore: could not read Modern Robotics USB DC Motor Controller [AL00XSUW]: comm timeout
                    03-04 14:24:19.360 2557 3000 W RobotCore: [55 aa 80 40 1e] -> [00 00 00 00 00]
                    03-04 14:24:19.360 2557 3000 W RobotCore: could not read Modern Robotics USB DC Motor Controller [AI041UCD]: comm timeout
                    03-04 14:24:24.330 2557 3037 E RobotCore: OpMode 'Main TeleOp' stuck in loop(). Restarting robot controller app.

                    At this point, the phone appears to be in an infinite loop trying to restart. Phone must be rebooted.

                    Best,

                    Chuck

                    Comment


                    • #11
                      Originally posted by FTC4318 View Post
                      The protective mat over the gym floor, coupled with the FTC mats, created an environment rich for ESD events. It was heartbreaking to see so many dead bots in both divisions. Talking to teams, and our own experience was that bots ran perfectly fine on the practice fields where there was no underlay. Go out to the tournament fields and robots experience strange, not repeatable errors when contact was made with other robots, the metal ramp or the field perimeter. End result is a dead robot.
                      Looking at the competition fields, it appeared that they used brand new foam mats. In contrast, the practice fields appeared to have used foam mats that were likely used throughout this season; which means that the mats were possibly sprayed with anti-static spray at previous events.

                      If what another poster said is correct, the spray would be effective for up to 30 days after application.

                      Maybe the difference between the practice fields and competition fields had more to do with that than the underlay on the gym floor? Which would then really call into question the tournament directors' decision not to use anti-static spray. Just a thought.

                      Comment


                      • #12
                        Hi folks,

                        I am not a member of FIRST's staff, and definitely can't speak for the engineers or development team. As a result, everything below comes from me, my experience, and my nonprofessional opinions.

                        The transition from version 2.4 to 2.62 of the SDK is a good example of my own failure as an FTA to understand and communicate the changes with teams. The changelog in the GitHub repository's readme says "Changes to enhance Modern Robotics USB protocol robustness." I know -- from official discussions, unofficial chats, and (later) research -- that this actually refers to a vital improvement in the SDK which my solve most to all of the Stuck in Loop errors many teams experience. The importance of the change was very clear to me, but I absolutely failed to communicate this information to a number of teams in my region who could have benefited. That's definitely something to improve next season.

                        I recommend a healthy level of caution on this topic, as it is very easy to perceive ignorance. I say this because several times during this season, I've been absolutely sure that a certain factor (including ESD) is the cause of poor robot performance, and also been absolutely wrong. Nobody here needs to be reminded that there are a multitude of variables at play when it comes to any given robot issue, but it's always healthy to frame the discussion in that light.

                        Okay, with that prologue out of the way, let's rapid-fire through some technical stuff:

                        -- Version 2.62 of the SDK fixes a USB communication issue which can cause the watchdog timer (a security feature to prevent robots from running wild) to timeout. Such a timeout can cause a Stuck in Loop (or similar) error irrespective of the team's code or ESD events. Please do upgrade, do test thoroughly, and do submit feedback if you find issues. It's probably still in beta because the changes are mission-critical and deserve testing. Deploying changes to the USB communication protocol to several thousand teams without testing could be dangerous.

                        -- With that said, version 2.62 does not magically cause the Stuck in Loop error to disappear, because it isn't an error. It's a safety feature, and a symptom of something else. If the robot's program hangs for five seconds, it's a good idea to have the robot stop. Needing to reboot the phone afterwards is a separate problem, and worthy of an issue on GitHub should anyone be willing to make one.

                        -- It's hard to replicate ESD in a consistent fashion. If you have access to an ESD "gun", there are a few people (myself included) would love to talk to you. You should not do what I did, which is to buy a $10 piezo grill igniter from Lowe's and send shocks across the frame of a non-competition robot.

                        -- ...but if you do, you'll likely find that the shocks cause the Stuck in Loop error pretty consistently. (insert grimacing emoji here) But, that doesn't necessarily tell us anything. Any given grill igniter may or may not be anything like an ESD event in terms of voltage or current.

                        -- At my events, I recommended spraying with water for a few reasons: (a) the anti-static spray does change the consistency of the tiles, and I didn't want to do that to your autonomous routines so late in the season, and (b) at least one person reported properly-applied water to be more effective anyway. However...

                        -- Even if you do spray with water, there are a lot of variables to consider. One is quite simply the weather, as you know that humidity affects the frequency of ESD events. The frequency of spraying is even dependent on how new the tiles are as newer tiles don't necessarily "soak in" the water and create a conductive plane in the same way old tiles do. That was alluded to above, but I want to drive home the point that we have seen success when using water "properly" under non-apocalyptic conditions.

                        -- I think FTC4318 did a good job of explaining why grounding straps aren't used (or allowed).

                        tl;dr ESD is hard; please be patient; use your local FTA as a resource. We genuinely care about teams having a good experience at events, and we are in the habit of continuous improvement.

                        AJ Foster
                        Just an FTA
                        Orlando, FL

                        Comment


                        • #13
                          Grounding straps don't work on an insulating foam mat, but they are very effective on a conductive one. They will be very effective if combined with either the ESD spray, spraying water on the mats, etc...

                          Comment


                          • #14
                            This is actually FTC4318. The forced reset of passwords for the forum has caused a problem with my account. I could only give a new password to this defunct team number that was associated with the same email address. Okay - here is the bottom line as I see it on this issue:

                            Confirmed - Not all failures during a competition are ESD related
                            Confirmed - ESD, when it occurs, is a serious problem for FTC robots. It can cause failures during competitions that are hard to replicate.
                            Confirmed - The mats are vinyl and many of the FTC approved wheels are nylon. When these two materials are rubbed together they become electrically charged (see triboelectric series) putting them at risk for ESD events.
                            Confirmed - A robot that has reasonable protection from charging up (ie, natural rubber wheels) can still be damaged if it comes into contact with a robot that has charged. An ESD event will occur due to the voltage difference between the two robots, i.e: the electrons jump.
                            Confirmed - Repeated ESD events can cause latent damage to electronics.
                            Confirmed - Anti-static sprays will greatly mitigate ESD events

                            Given this knowledge it is common sense to reduce the chance of ESD in FTC. The most reliable and cost effective way to do so is with anti-static spray for industrial environments (as opposed to "static guard").

                            We use anti-static spay on our field once every year. It does NOT cause our robot to move any differently than it would not a non-treated field. We work in a barn that is only heated during our meetings. The heat source is a pellet stove. It produces a very dry heat and runs for hours to bring the temperature up to a comfortable level. In short, we must spray our field once a year in order to avoid continual shocks while interacting with our robot. The spray works beautifully. When you overspray then it can cause build up. Don't over spray and you won't have that problem.

                            As stated before, we competed at WVA Championships this year. They treated their fields with anti-static spray. No one had issues with their autonomous or anything else related to how their robot was moving on the treated tiles.

                            Note on repeated ESD events can cause latent damage to electronics - there are reports of fried Andy Mark beacons this year. The company did design them to withstand ESD but did not or could not protect them against repeated ESD events. How can we ask teams to do so when building their robot? What about all those teams who compete for months with the same electronics only to have them fail at the higher levels of competition due to latent damage caused by cumulative ESD events? Are they just supposed to replace all their modern robotics components and other electronics before every competition despite the steep cost?

                            Professional engineers would never attempt to operate electronics in these conditions without some sort of mitigation. Why are we asking teens to do so? What are we teaching them? Shouldn't we be modeling good engineering practices?

                            Comment

                            Working...
                            X