Time to re-think ESD (electro-static discharge) strategy
I don't fully understand how FTC settled on the current ESD rules and field elements, but I can well imagine it was a well intention-ed attempt at minimizing robot-to-robot zaps. The policy is one of isolation: keep all robots isolated from each other as much as possible. Industry does the opposite: tries to keep everything at the same potential to prevent build-up in the first place. Let compare the two methods:
-> All objects try to ground themselves using a high impedance to a common ground
-> If possible, increase the air conductivity by keeping humidity up
-> Use air ionizers if possible to help discharge surfaces that aren't grounded
-> No one is allowed to ground themselves
-> All conductive materials need to be protected/kept away from touching foreign conductive materials
-> Very careful selection of materials in your robot needs to happen in order to minimize charge creation
-> All metallic objects (e.g. Robots) become big storage vessels of energy.
-> Energy builds up and up until the rate of discharge through the air (thanks to moisture) is able to keep it from building up further
-> The mats are non-conductive and thus all friction with them easily creates energy, making the build-up of energy very easy
-> Anything that is not well insulated (other robots, beacons, etc...) are now vulnerable to damage
-> Electronics are not built to handle this massive amounts of energy, thus if they are zapped (e.g. beacons) they will have damage. Most micro-chips are built to handle <= 2kV zaps (human-body model), which is a level you don't feel. The voltage levels you feel are >= 10kV!
Trying to solve ESD issues becomes very difficult because it is largely dependent on how much moisture is in the air. People in Florida may have little or no issues ever, while people up north in the cold winters will experience lots of issues especially during cold-spells.
One of the mis-conceptions people may have is that the act of grounding the robots will result in robot-to-robot ESD events. Grounding everything does NOT mean creating low-impedance paths.. quite the opposite, ESD grounding involves very high-impedance conductive surfaces (e.g. 1 Meg Ohms). The purpose of grounding is trickle-off built up charge, not to rapidly discharge it.
I propose FTC embrace the following policy changes. As FTC grows year to year, these will get harder and harder to implement and thus the sooner FTC bites the bullet the better:
1) Change the mats to ESD compliant ones! These mats are the key instigator of easy ESD build up. ESD compliant ones will help discharge build-up and help prevent build-up in the first place.
2) Allow robots to ground themselves to the mat. An ESD grounding strap that is high impedance (and made of soft fabrics) will neither cause damage to the field, nor create low impedance paths for robots.
FTC has proposed spraying the field with water and/or using an ESD spray on the field. While these quick-fixes are helpful, they change the field's characteristics (friction) and while help, do not prevent the problems as effectively as adopting new mats and policy changes.
Couldn't agree more with those suggestions. Spot on. One addition would be ensuring the mats are appropriately earthed either via wired connection (compiant to regulations of course) or ensuring tiles are placed on a charge dissippating surface (i.e. not carpet!)
Originally Posted by gszczesz
Anti static sprays - their proper use in FTC
I agree the ideal solution to this problem is to replace the FTC mats. However, conductive mats are expensive.
Originally Posted by FTC5110
At the very least anti-static spray needs to be used on all competition and practice fields. We just came back from East Super Regionals. The organization and volunteers running the event are top notch. We very much appreciate their hard work and dedication.
Unfortunately, however, the decision was made collectively by ALL regional directors to NOT use anti static spray on the fields. East Supers did spray water every now and then but not according to the recommendations from FIRST, ie: dampen the field and re-apply every 10 minutes.
The protective mat over the gym floor, coupled with the FTC mats, created an environment rich for ESD events. It was heartbreaking to see so many dead bots in both divisions. Talking to teams, and our own experience was that bots ran perfectly fine on the practice fields where there was no underlay. Go out to the tournament fields and robots experience strange, not repeatable errors when contact was made with other robots, the metal ramp or the field perimeter. End result is a dead robot.
The recommendation from FIRST on how to use the anti-static spray is not correct. I spoke with the manufacturer of the Tech-Spray Anti Stat 1726QT. Tech-Spray recommends applying their product at the most ONCE every 30 days for DAILY USE in an industrial environment. FIRST says you have to use it before every competition which is NOT recommended. That causes a build up that effects the tile surface. Eventually the mats will need to be washed or replaced as they become too tacky.
We saw many discouraged teams and kids at East Supers because of the ESD issues. Even the number one seeded alliance in the Tesla division got zapped by this during the semi-finals. This should not be happening. These teams worked incredible hard to make it that far in the competition. We've not seen anything like the number of dead bots in other competitions and these teams would not have made it so far if they had not done everything reasonably possible to build and program robust robots.
I urge FTC to revisit this issue. Get the manufacturer of these highly engineered anti-static sprays involved in the solution. They most likely will help set firm guidelines for everyone's benefit.
I would have to strongly agree. At our local competitions we have not seen a lot of ESD issues because the FTA's have either treated the fields before the event with Staticide or are making sure to water down the field around every 10 minutes or so. The only time we have seen issues is when we neglected to wipe our own robot down before matches.
That being said we were just informed from the North Super Regional that they will not be using any type of ESD prevention at all. This is very frustrating as a coach to see teams struggle and see all of their hard work dead on the field when the solution is put out there for all tournament directors. In the tournament director manual it even states that the best solution is to use an anti static spray and then to use water if anti static spray is not an option. To a point I do understand the ruling from FIRST in the official forums that they are leaving the decision to use ESD prevention measures up to the tournament directors since some areas are more prone to ESD than others. However, it is very frustrating when our local and state tournaments are taking more care to prevent ESD than our Super Regionals.
I hope in the future they do address this and try to set some sort of standard that all tournaments should be following to help decrease ESD events.
Something does need to be done about the ESD issue.
I completely agree with this. With the number of hours that goes into this endeavor and to have an ESD issue destroy it is criminal. I can't understand why FTC continues to ignore this problem. My mats measure -6000 volts all the time. I have used commercial anti static spray and it works amazingly well. I think that the least expensive solution would be to conduct tests with a spray and then declare it as the official spray to be used at all tournaments. One bottle of spray cost $20, and in bulk could be less. I'm sure that all coaches and mentors would gladly pay an extra 1$ in entry fees to have the fields sprayed. If teams knew that a certain spray was going to be used all of the time, then they could test with it on their home fields to make sure that the possible change in field friction was designed into their robot. I'd be willing to conduct the tests myself. Living in Colorado makes it easy to find low humidity days to test solutions to this problem. I encourage other coaches and mentors to speak up on this issue. We (teams under the FIRST umbrella) should have some say on this matter.
Frank Bernett (Mentor for Up-A-Creek team #11260)
There was mention above about the effect of the protective covering between the floor and the mats in the competition fields. The layer is typically not present on the practice fields. Does anyone have an idea about how much worse this makes the problem? If someone has the tools to measure the mat voltage, would they be willing to check this with and without the additional layer (tarp or such)?
Given the foam mats are ideal insulators, what happens directly below the mats shouldn't affect anything.
Originally Posted by FTC7253
On the other hand, the floor around the field acts as a grounding point for people and the moisture in the air, thus if it is a conductive surface it helps prevent buildup while carrying the robot on/off the field. Consider if the field was on carpet, then every time a person walks to the field and touches the metal wall they'd be charging the wall up, and the robots would have no way to equalize to that charge level except when they touch it.
I wholeheartedly agree with idea of standardizing on anti-static spray, as a bare minimum.
Originally Posted by FB_BF
At our state tournament, the fields were not sprayed and the results were disastrous for my team and several others. Our robot died in 4 out of 5 qualifier matches with the dreaded "OpMode stuck in loop()" error. This error had not appeared in the four competitions we played and many hours of practice leading up to the State Championship. It caught us completely off guard, despite having treated our own robot with anti-stat spray and insulating exposed metal. It was the most frustrating experience in my five years mentoring. Our team worked harder and smarter than ever before, performed exceptionally at every meet and tournament to date, and had high hopes to advance to Supers. Those hopes were dashed in their first match when the robot didn't get 10 seconds into TeleOp before locking up completely.
Many other robots battled the problem throughout the day. In 37 matches I watched, 20 of them had at least one dead robot by the end. I can't say they were all from ESD, but I'm betting the majority were. The FTAs told me they were getting shocked frequently when they touched a robot.
My other thought on this is that rule on grounding straps should be flipped. If they really make a difference, they should be strongly encouraged. Teams that don't ground should be warned that they run ungrounded at their own risk.
This problem must be solved by FTC. To ask the kids to work on ESD prevention, which is nearly impossible to replicate reliably, is asking too much. As I said, we didn't see it until State this year. The kids should be thinking about their approach to the game, designing and building, and everything else FIRST encourages - not figuring out all the "tricks" you need to do to prevent shocks. ESD needs to be out of this equation. That being said, we've left our robot intact for the post-season, continue to try to replicate the problems, and devise and test static mitigation strategies. Not having a lot of luck on the replication part though. I'm currently in the market for a static gun I heard about somewhere.
5119 Baryons Mentor
Hi Chuck - I am so very sorry 5119 had such an experience. Here are a handful of thoughts:
Re: grounding straps. Unfortunately, grounding straps will not work. The physics is counter intuitive, but the mechanism by which a robot charges via ESD is not the reversible via a ground strap. The competition tiles are highly resistive. They will not easily accept current charge from the bot. The mechanism by which they charge is electro-mechanical. So while they seem attractive, grounding straps don't work and only pose a entanglement hazard.
Re: static gun. There are such things and they can be rented or purchased for a hefty price. They can also be built - but you would still need the power supply. If I was at work, I could send you the test equivalent circuit for a standard discharge. I believe first did a fair amount of testing with MR hardware. If my memory serves me correct, Thomas Eng spoke about a 6% failure rate (I need to look for the old post). I do not know what the test conditions were nor the hardware under test (E.g.: phones, etc).
Re: OpMode Stuck In Loop Error. Please check your phone's log files to see if there is any anomalous things going on. There are reasons why the phone would exit (e.g.: infinite loop) that have nothing to do with ESD.
Again, I think that though this problem can't be eliminated (except at huge cost), it can however be substantially mitigated by the use of anti-static sprays - - used according to manufacturers directions. FIRST had it right last year, but went too far with over application. It gunked up the fields and now they've swung too far in the wrong direction - not using at Supers. The sprays were used correctly at WVA States and I can't remember a single electrically dead robot. So, in my mind, the logical place to go is for FIRST to highly recommend the proper use of these sprays.
And while I am on a roll - I believe water to be a very poor substitution for anti-static spray.
- Water is highly variable in its conductivity. Distilled water is very non-conductive. Add some ions and it becomes semi-conductive. So right off the bat, you don't know what you've got.
- Water has high surface tension. It likes itself better than it likes the mat and is more interested in making quasi-non-deterministic semi-conductive islands than it is in making a true conductive film on the mat.
- When the water evaporates - it's done. When it is partially evaporated, it's also done, since any conductive surface has now been broken up as the water evaporates unevenly.
Sorry to be long winded - but I am equally displeased with the performance of this hardware in a high ESD environment. However, since the hardware is what it is and needs to be affordable, the only thing left to do is to is change the environment.
Thank you for your thoughts, FTC4318. I suspected grounding straps might be a bust, but if I can replicate the situation in a controlled environment, I might test it out anyway.
Regarding the logs, I did collect them from the phone after the tournament, we did see three motor controllers timeout simultaneously. I'll paste the relevant bit below.. I'm not sure how that could happen besides ESD or loose wiring (we had good, strain-relieved connections and new USB cables with well-fitting connections). Also, the fact that I haven't been able to reproduce this since the weather got warmer and I haven't received any shocks during testing...
I know that there's a newer SDK to help with the timeouts, but we hadn't installed it before State, since you generally don't want to fix something that ain't broke. We'd been stable up to that point. Again, if i can reproduce, we'll test this as well. Waiting for another cold snap here in Missouri so I can get some good dry air in the house.
It's also possible we have something flaky in our PDM, being the common component to the three simultaneous timeouts. It's a complete mystery to me so far, and, admittedly, static is the easy explanation. When we run this in a humidified environment (my home or at school), we don't see the OpMode stuck in Loop(). Perhaps something in our setup is more susceptible to ESD problems. I just don't know what it is.
Here's the log from our last match of the day:
03-04 14:24:19.320 2557 3245 W RobotCore: Modern Robotics USB header sync bytes are incorrect
03-04 14:24:19.320 2557 3245 W RobotCore: [55 aa 80 40 1e] -> [00 00 00 00 00]
03-04 14:24:19.320 2557 3245 W RobotCore: could not read Modern Robotics USB DC Motor Controller [AL00VF7C]: comm timeout
03-04 14:24:19.360 2557 2974 W RobotCore: Modern Robotics USB header sync bytes are incorrect
03-04 14:24:19.360 2557 3000 W RobotCore: Modern Robotics USB header sync bytes are incorrect
03-04 14:24:19.360 2557 2974 W RobotCore: [55 aa 80 40 1e] -> [00 00 00 00 00]
03-04 14:24:19.360 2557 2974 W RobotCore: could not read Modern Robotics USB DC Motor Controller [AL00XSUW]: comm timeout
03-04 14:24:19.360 2557 3000 W RobotCore: [55 aa 80 40 1e] -> [00 00 00 00 00]
03-04 14:24:19.360 2557 3000 W RobotCore: could not read Modern Robotics USB DC Motor Controller [AI041UCD]: comm timeout
03-04 14:24:24.330 2557 3037 E RobotCore: OpMode 'Main TeleOp' stuck in loop(). Restarting robot controller app.
At this point, the phone appears to be in an infinite loop trying to restart. Phone must be rebooted.