lidar Lite Issue

[dcwalmsley]

Need some help.


Last week I proceeded to bench test the LIDAR unit I just received and came across a peculiar issue.

On several bench test I discovered that the LIDAR unit stopped sending data to the Pixhawk. First attempt showed it stopped around 5.5 minutes, second test about 1 minute, third test was around 8 minutes, etc...
Further test showed it wasn't the Pixhawk which was the issue as resetting the Pixhawk didn't show it receiving data from the unit. So I left the Flight Controller (FC) connected and monitored data and pulled power plug from LIDAR unit and then reattached it and the FC started seeing altitude data again. This was done several times with same result.

So at this point, I think there is a communication issue between LIDAR Lite and Pixhawk.

I have left my concerns with Pulse Light on their blog in an attempt to find a solution to this problem. See. https://groups.google.com/forum/#!topic/pulsedlight3d/JOjvyywFyWg

Please take a look and tell me if I'm not doing something right. I will post my configuration settings this evening and latest log as well.

Thanks,
Doug Walmsley

[Randy Mackay]

Doug,

Ok, so according to Lidar-Lite they think it's an issue with the Pixhawk side driver not waiting for an ACK or NACK. Issue raised below.
https://github.com/diydrones/ardupilot/issues/1770

-Randy

--
You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Doug Walmsley]

Thanks Randy.

~Doug W


On January 12, 2015 10:46:20 PM "'Randy Mackay' via drones-discuss"

[Lorenz Meier]

Hi Dennis / Randy,

I did discuss this with Dennis earlier and I’m not sure about the presented conclusion. In particular that power-cycling the sensor alone brings it back to live is worth a scope / logic trace of the bus, to really understand what is going on. The conclusion that the host is the problem doesn’t seem quite obvious yet.

I also looked at the PulsedLight code example, which is polling at 1 KHz for the sensor to send an ACK response, and only then reads it. At least in the example code this means that a majority of the bus bandwidth (in particular when you take the overhead of each transfer into account) is consumed for communication attempts which do not even transmit any data / payload.

It would be nice if you could provide a tested code example with a more realistic communication scheme that is implementable on a real system. Right now I still don’t understand the ACK/NACK argument and I believe you didn’t mean to imply that the host has to poll at a kilohertz in order to establish reliable communication.

I do have more feedback on the communication interface in general (which both Tridge and I shared already earlier), but I don’t want to clutter this thread and would like to help to work out this particular problem first.

Cheers,
Lorenz

[Dennis Corey]

Hi Lorenz,

I think we really need to do a reset on this particular issue.  The core problem as I see it is that the Pixhawk code was written to work with a sensor firmware version that is now quite old.  We also believe that the current Pixhawk code is implemented based on a fixed timing scheme that was necessary to work with these early versions of the sensor firmware.

Also, the sensors that most of you have at 3DR are mostly preproduction units that, while they have firmware which is newer than what was used for Pixhawk driver development, are still not the current release that we are shipping.  These facts on their own are certain to lead to some interesting results. :-/

The sample code we provide on our site is merely to allow our users to get their sensor up and running as quickly as possible and provide them as the basis for additional code development for their projects.  And you are correct that we are by no means suggesting that kilohertz polling is necessary for stable operation of the sensor.  We do know from experience that any fixed timing scheme to read and write data is bound to fail as our device has variable acquisition times that must be accounted for in any code developed to be used with LIDAR-Lite. 

So our conclusion that Pixhawk drivers need updating are not to place blame but simple observation based on the known development history.  We are already talking to Craig about a collaborative effort to resolve the issues and provide users a great experience.   I'll also add that there are many functions of the sensor that are not being utilized and I hope that through our collaboration we can also implement some of these other capabilities to make the user experience even better.

As a first step we need to at least get everyone in the 3DR development teams running the most current revision of LIDAR-Lite firmware.  So we need to get those older units back and updated. (Sorry, this is not a field upgrade - yet.)  Once that's done we'll all be working from the same base firmware.  In the meantime, we'll be working with 3DR on the Pixhawk updates.

Dennis

[Philip Rowse]

Thanks Dennis :)

Philip Rowse 

Lead Systems Engineer

3DRobotics

Ballarat

Australia

[Lorenz Meier]

Hi Dennis,

Thanks for the update. I still would love to see deployable example application code not keeping the bus busy, but instead doing 10-100 Hz readouts. There is nothing more accurate in terms as interface description compared to a working example.

How can we tell which Firmware version on the sensor are we talking to? Can this be read by the driver or identified by a label on the module?

Cheers,

Lorenz

[Dennis Corey]

You can tell hardware and firmware version by interrogating registers 0x41 (hardware) and 0x4f (firmware) the current revisions will read out 0x17 for hardware and 0x3 for firmware.  I doubt the units you have even have those registers active.  If they are I can almost guarantee that they will read out 0x01, 0x01...  I'm afraid the manufacture date label is going to be of any use.  

Dennis

[Lorenz Meier]

Thanks Dennis.

The code example would still be very much appreciated.

-Lorenz

[Dennis Corey]

Austin and I talked with Craig yesterday about doing some work on the Pixhawk drivers.  I'm sure something will come out of that.  Since we're not the Pixhawk experts we'll be making recommendations and providing samples to the team of what we think the driver will need to look like so that they can dot the "i's" and cross the "t's" on the Pixhawk side.

[dcwalmsley]

Today I updated my pixhawk firmware to v.3.2.1 and took the OctoQuad w/Lidar up for a short flight and have logs for you to review.  

Hope the logs will help.

https://www.dropbox.com/sh/78sgjdiepr84zru/AAD59SqDT51AAWTiGbB_dK8Ta?dl=0

R,

Doug Walmsley

[dcwalmsley]

I'm writing this here to ask DEVS for answers to the development of drivers for Lida Lite. Since I have reported in Drone-Discuss site back in January, I have noted that the energy to fix the problem has wained. The manufacture indicated they are actively work with 3DR to fix the 1st generation units that have been shipped with new drivers which allows Pixhawk to maintain ACK with LIDAR Lite module that randomly drops offline.

Recently I went to PulsedLight's website and had a difficult time locating the forum page but was able to find it via Google search engine. However, once I entered their forum site, I realized they are no longer accepting blogs/messages to developers. You have to leave a ticket or read over possible solutions. Disappointment is an understatement. At this point, I do not see a any momentum in resolving the Lidar Lite / Pixhawk communication issue. 

I like to use my two LIDAR units, but feel the only way to resolve this is to ship the modules back to Pulselight to have the firmware flashed or ask for a refund. 

Craig, I'd appreciate some feedback as to where things are in the fix process and when it will be implemented?

Thanks,
Doug W.

[Philip Rowse]

This issue is actively being worked on, with Tridge doing experimental work on receiving the Lidar data via PWM.

Sent from my iPhone

--

[Tom Pittenger]

Doug,

The PWM driver is complete and works great. I2C driver bug is still broken and cannot be fixed by APM although there is a workaround that will allow the APM to detect a locked up lidar Lite and peform an external reset it but I don't think that is tested whatsoever. The main focus was to have PulsedLight fix the I2C while Tridge implemented the PWM driver to support existing units.

-TomP

[Jesus Alvarez]

I do have two units too.
I was going to use them in my two planes but heard about the I2C issue. As this bus is critical I think I will wait for AC 3.3 and use PWM.

However Pulsed light should give us a way to upgrade our units. Right now they are a little useless.

[Tom Pittenger]

As Jesus said, the PWM driver is in 3.3.0 beta now. here's the details:
http://diydrones.com/profiles/blogs/apm-plane-3-2-3-and-3-3-0beta1-released

[Doug Walmsley]

This is good news indeed but won't help me or Arducopter folks.

I certainly don't have a problem rewiring/configuring my LIDAR Lite unit but the question now is when will Arducopter get this work around?

[Dennis Corey]

The PWM driver is a nice alternative to I2C and has the added benefit of being able to power off the sensor when flying beyond the maximum range of LIDAR-Lite to save power.  Having said that, PulsedLight has also been working on resolving the I2C glitch and will be sending updated units to 3DR next week for testing and preparation of a driver update.  For customers that have LIDAR-Lite sensors, we will provide this update at no charge and new sensors that are purchased from us or our distributors will have this fix already in place.  To receive the update you will need to send your sensor to us and we will return it to you promptly.

[Philip Rowse]

Feel free to port it :)!

Sent from my iPhone

[Philip Rowse]

Thanks Dennis!

Sent from my iPhone

[Craig Elder]

Thanks for all the great responses.  Not too much I can add here Doug

The PWM interface is described here.  It will be released as part of copter 3.3 or you can fly it in Master at your own risk

http://copter.ardupilot.com/wiki/common-optional-hardware/common-rangefinder-landingpage/common-rangefinder-lidarlite/#connecting_to_pixhawk_via_pwm

It is already available for plane and rover.

Most likely by the time Copter 3.3 is released there will also be a fix for the I2C driver to work with the revised Lidar Lite hardware as well.

[Doug Walmsley]

Sorry but I'm not a programmer and never learned how to do it.

~Doug W

[Federico Mari]

Just got this form Pulselight 

"" The “glitch” in the I2C bus that was causing problems with the Pixhawk autopilot has been fixed and the units we are shipping have that fix in place along with the filtering on the I/O lines in and out of the sensor.  We’ve also made some other improvements to the firmware that have improved accuracy and fixed a couple of other little bugs.   3DR has not had the opportunity to test the fix, but we anticipate that there will be no other issues and that they will release a Pixhawk driver update soon. 

""

Great news

[Graham Abell]

How do people who've already received their units (like me) get the new firmware?

[Josh Welsh]

I know the process is “send back to PulsedLight” but yeah, I’d like to know the specific steps as well so I can get two sent back myself.  Is there a form we fill out, someone we email?

Thanks!

--

[Robert Lefebvre]

Ditto.

[Doug Walmsley]

+1

[Austin Meyers]

Hello all, 

LIDAR-Lite requires a chip and hardware update to correct the I2C glitch and add filtering. Once 3DR has had an opportunity to test and release a driver update we will advise current users on how to get updated units.

- Austin, PulsedLight

[DB]

Thanks for the advice. Got 2 from SF (so far) and so holding my breath just like everyone else.  Best. Daniel

[dcwalmsley]

Well we're rolling into April and still no status or fix from anyone on the 1st gen Lidar-Lite modules. Appreciate if someone can provide feedback of when we can see a fix? Weather is improving and I'm itching to fly with the sensor soon.

Thanks.

~Doug Walmsley

[Dennis Corey]

Hi Doug,

You can fly the sensor now using the PWM method as described in the Arducopter wiki, http://copter.ardupilot.com/wiki/common-optional-hardware/common-rangefinder-landingpage/common-rangefinder-lidarlite/ .  This works well and has the added benefit of powering the sensor off to save battery when you are beyond it's maximum range.

Tridge and Craig Elder had a chance to test the new sensors late last week and the preliminary results were positive.  The general sense is that the hardware and firmware fixes we implemented have solved the problems that were being experienced when connecting to the Pixhawk.  I know they would like to do further testing before giving the final stamp of approval and we think it is still a good idea to wait just a little longer before moving ahead with updates until we know with certainty that what we've done has fixed the problems and not created any new issues.

Anyone who has a sensor with a February 2015 manufacture date label already has a sensor with the fixes in it that were tested by Tridge and Craig last week.  Assuming that nothing else needs to be done to the sensors, those customers will be ready to fly as soon as we get an official sign off.

Sensors manufactured prior to that date can be operated via the PWM mode, but will need to be updated if they want to connect via I2C to the Pixhawk.  Since this is both a hardware and firmware upgrade, we think that the best upgrade option, especially for international customers where shipping costs are a factor, may be to offer a discount toward a new sensor that includes all the fixes.  We will let everyone know how this is going to work soon.  

So the news is good!  We have made a lot of progress and the early results look very promising.  We'll keep you posted...

[Tom Pittenger]

Thanks for the update Dennis! Does the new firmware address any GPS jamming issues that have been reported?

--

[Doug Walmsley]

Sounds good so far.  Do I need to add the capacitors across the wiring harness or can I use the harness as shipped?

--

[Dennis Corey]

The firmware doesn't address the GPS issue, but the full PCB ground plane, filter capacitors and inductors built into the board do and that's the important part of the upgrade for UAVs.

[Tom Pittenger]

so the "new" Lidar Lites are not just a firmware update, they're a new PCB as well? 

[Dennis Corey]

For the PWM application you can use the harness as shipped no filter caps needed on all versions of LIDAR-Lite.  For the I2C application the filter caps are now built into the board as of the Feb 2015 build and the harness as shipped will be all you need.

[Dennis Corey]

Yes.  And a change in the FPGA we use for the signal processing.  Again, all included in the Feb 2015 build.  There's been a lot of work behind the changes and it's virtually a new product.  That's why we really want to make sure there are no other issues before moving forward with updates.

[Srikanth Saripalli]

I have been using the new Lidar lite in Alt Hold mode using I2C and it works flawlessly

-Sri

[Tom Pittenger]

Due to GPS jamming problems, I am powering down the LIDAR until I start my landing approach via gpio. Is that something I can do via i2c?

[Srikanth Saripalli]

No you cannot. I haven't had any GPS problems though, but I am using a different GPS unit (Ublox M8Q)

-Sri

[Dennis Corey]

The sensor has the ability to be put into a "sleep" mode via I2C, but this would require the Pixhawk driver to implement that feature.  The power enable line of the sensor can be used to power it on and off in the PWM mode, but I'm not sure if the RNGFND_STOP_PIN can be enabled in the Pixhawk when the sensor is being used via I2C.  Perhaps one of the developers can comment on that.

As far as GPS issues those should be mostly resolved now.  We do still recommend powering the LIDAR-Lite form the BEC and we also have found that a simple foil shield between the GPS receiver and all the electronics can improve the situation significantly.  Keeping the leads between the sensor and the power and Pixhawk connectors as short as possible as well as clipping off the unused sensor leads can make a difference.  A clean installation makes a difference.

I think we need to start tracking what brands/models of GPS systems do or don't have problems as that might point to some other issue we are not aware of.  As I've mentioned before our testing has not seen any GPS issues, but then we've only been testing with the stock GPS as supplied with an IRIS and also one that came with APM system that we've plugged into a Pixhawk.

[Robert Lefebvre]

I've definitely seen a lot of anecdotal evidence that the 3DR GPS is more resistance to interference than some of the alternative vendors.

[Dennis Corey]

And as we start putting more sophisticated sensors and other devices on our drones this is going to become a pretty important issue.  I know that the IRIS drones now have shielding for the GPS since the receiver is directly above the electronics and I have seen other drone manufacturers resort to common mode ground and Faraday cage implementations in their products.  It would be great to see a side by side comparison test of the different GPS systems under different operating conditions.

[Xerr Avon]

Hi,

I have a Lidar-Lite with the Feb 2015 sticker on it and am having problems with it on a pixhawk running 3.21. On the I2C line it will run at times but then lock up, and takes several connect and reconnecting the battery to get it to run. I see in the wiki it says units made before March 2015 have the I2C problem but here it is saying the Feb sticker units are ok? I am sure that my Feb 2015 unit is not working.

thanks,
John

[Dennis Corey]

The wiki is being updated, but every unit that has a February 2015 manufacture date has all the updates, so you can be sure that your unit is the latest.  There is a note in the wiki indicating that the BEC ground (it's recommended that the LIDAR-Lite be powered from the BEC) should also be connected to the servo ground rail.  Power noise and other installation variables can cause problems with the I2C communications.  We are also working with the Arducopter development community to make sure that there are no driver issues that may be affecting I2C communications.

[Xerr Avon]

Hi,

I have the grounds connected together and am powering from a SBEC from a ESC.  still It is locked up more than working. Once I even had it cause other problems on the I2C line, my external LED would not turn on and the compass readings were off. not sure if it was locked up or not. as soon as I unplugged the lidar everything was fine. not sure what to do.

John

[Dennis Corey]

If you're up to some experimentation I would try disconnecting the other devices on the I2C line and see if the LIDAR-Lite still locks up.  It's possible that there is a conflict with one of the other I2C devices address.  If it works after disconnecting the other devices, start adding them back onto the bus one at a time and see when the problem reappears.  If running the sensor on the bus by itself doesn't resolve the problem, the next step would be to confirm all of the configuration settings in Mission Planner.  And finally, try connecting a 680uf electrolytic capacitor across the power and ground leads of the sensor(observing polarity).  If none of those things work you could possibly have a defective sensor.  We're more than happy to have you send it to us to be checked out and replaced if necessary.  I also wouldn't rule out driver issues, but so far the reports from the developers are that the sensor works just fine with the current drivers.

[Xerr Avon]

Hi, I will do some more testing tomorrow but from the testing I have done so far it appears it could be my external compass though I am sure I tried this a couple days ago and still had it lock up with the compass removed but for now I cannot get it to mess up without the compass plugged in. The thing is I am using an M8N GPS/ with magnetometer I dont believe there are a lot of people using this unit. I will try using an 3dr magnetometer with the M8N (I love the M8N GPS cannot give it up) and see how it goes. I am pretty sure it is the magnetometer as it locks up on startup many times with just the lidar and compass installed. I also have an LED and PX4Flow installed but I am not currently using the PX4Flow nor am I using the FW for it so I will leave it disconnected for now.

Is there any way to fix the problem with the M8N's compass (if it does turn out to be the problem)? I will report back when I am confident it is or is not the compass.

thanks,

John

[Xerr Avon]

Hi,  I did some more testing, the lidar seems to work fine with no other devices plugged into the i2c port but as soon as I plug in any of the compass's in it either does not appear to start (reads 0), locks up and gives bad readings (10.25 and I have seen 28.33 on bench it usually reads .21) on lockup.  The compass/magnetometers I have tried is the one on the M8N GPS, a 3DR magnetometer, and a 3DR GPS/Compass module. all give me similar problems. not sure what else to try except for PWM but I am not crazy about flying Master for a while (too many recent changes I have not caught up on reading through the threads yet)

Oh, also I very much doubt it is a problem with the lidar as I have had this one replaced (from sparkfun) once because I thought it was the FW problem as the wiki said any before march 2015 had i2c problems, it gave me the same problems.

Only other thing, I doubt this will help, I have not tried this on a second pixhawk yet, I will do so tomorrow and let you know.

Just as I was finishing this message I thought I had not tried just the 3DR external LED and lidar, same problem, on start reads 0 and locks up. so it seems like anything else plugged into the i2c port with the lidar causes problems.

John


John

[Xerr Avon]

Hi,

I dont know, I have tried everything. thought I had it fixed a couple of times where it would run for close to an hour on the bench,  then fail, once it fails then it is even hard to get it to start running. sometimes it will run for a long time then give a bad (large number) then lock up. I have tried everything you mentioned plus tried a different copter/pixhawk. not sure what else to try. I may load master and try PWM tomorrow

John.

[Dennis Corey]

We've done a little thinking about the problem and we need to do a little more investigation, but it may be that as you add additional devices to the I2C line it is getting pulled too low to operate properly.  At the request of our users we have added filter caps and 4.7k ohm pull-up resistor on the sensor's I2C lines.  Not knowing if the Pixhawk has any pull-ups on the I2C or if any of the other devices you are using might have filtering or pull-up resistors, the solution may simply be to find a way to reduce the combined load on the I2C of all the devices.  Too much filtering or resistive load may be the cause.  

[Tom Pittenger]

Nodes shops not apply the pull ups, that should only be done by the master. Sound like the impedance is too low. Dennis, can you supply direction as to which resistors on the board are the pull ups so they can be removed?

--

[Xerr Avon]

Dennis,

I was thinking, I had ttried a bunch of stuff that at times I thought had fixed it, like I said. one example, I noticed on the copter I was trying to put this on for some reason I had all of the 5v lines from all 4 esc's connected so I pulled the 5v from all but 1. I thought this fixed it for a while, also for a while I thought it was the 5.6v zener diode but it was not. like I said it would run for a long time with no problems, this was with all devices connected to the i2c port. so today I ran it for a while with all other I2C devices disconnected. It ran great for a a long time but finally locked up. after it locked I tried to connect and reconnect the battery and was  sonarrange = 0.  One time I left it sit like this all powered and aparently locked up reading 0 a few mins later I looked up and it was again running. I have seen this happen a few times now.  so I am not sure it it is locked up or just not displaying the updates.

It is so hard to tell what is what it is very intermittent.  sometimes it is working and I cant get it to fail other times it is apparently failing and I cannot get it to work :)

Oh, one thing I had not mentioned, on the sonar voltage status item is always  "sonar voltage=0" should this have a value in it?

John

[Philip Rowse]

The Pix Hawk already pulls the bus up, this is a basic requirement for I2C.  Slaves should never have pull-ups.  Are they pulled to 5v or 3.3v?

Phil

--

[Tom Pittenger]

If we are powering the LIDAR from the ESC, it would be better to pull the i2c up via ESC but that is not practical. Sound like the i2c connection is hosed.

Glad I'm using PWM! On a side note, I had a problem the other day where my lidar didn't power up correctly, it was showing height values with a large offset. Would a lack of external DC bypass caps cause that?

[Dennis Corey]

Well, because of our eye safety certification I have to advise you that there are no user serviceable parts on the system and we're talking about almost invisible 0201 size parts anyway.  Secondly, in looking at the schematics for the Pixhawk, they are only using 1k pull up resistors and combined with our 4.7k the combined impedance is going to only be around 800 ohms so we don't think the pull ups are the problem.  Unless the info we're looking at is incorrect.

There are any number of possible scenarios that could be causing this problem and it's difficult to diagnose and resolve in this forum.  Our experience with the Pixhawk is very limited and we don't have the resources to commit to becoming experts on that system.  This is going to require someone with an oscilloscope, intimate knowledge of the Pixhawk system, a LIDAR-Lite sensor, a lab environment and the ability to consistently reproduce the problem in order to full understand what's happening.  We would be happy to collaborate on this effort - Ideally with a 3DR developer or engineer as it would be in everyones best interests.  Volunteers can contact me at den...@pulsedlight3d.com

[Tom Pittenger]

The problem is if powered by ESC as you suggest, you're pulling up to a different bus.

[Dennis Corey]

We have pull ups because not all devices that can be run as master have them and we try to be as compatible with as many devices as possible.  It hasn't been a problem on any other platforms - so far...  With the Pixhawk the combined impedance should not be a problem based on the info we are looking at.   

Our bus can operate at 3.3 or 5v based on the master bus voltage.

[Dennis Corey]

For now PWM is probably the best until we can work with someone at 3DR to fully understand the Pixhawk I2C interactions.

No, the power up problem could be simply that the power bus voltage ramped up slowly or there was a voltage spike/drop at some point during power up and that resulted in an bad sensor initialization.  Strobing the power enable line would reset the sensor and clear the problem.

[Tom Pittenger]

In my case, the enable line was holding it in reset and when it resumed is when I had the problem.

[john.in...@gmail.com]

I have been working with a PixHawk unit connected via I2C to a LIDAR-Lite unit.  I have seen the same issue.  We ended up dumping I2C and going to the PWM interface.

The LIDAR-Lite vendor has been contacted with various issues on the I2C interface.  They indicated that the units shipping in later March and onward would have a slight HW mod to suppress noise coupling to the I2C lines.  We attempted to make the same mods to 3 units we already had in hand with limited success.  The data drop-out was still observed.  We even had the units cycled through the vendor to be repaired.

The PWM interface is supported in the latest code for the PixHawk.  I suggest you use that instead.

As a side note, I have tried to get the PixHawk/ArduCopter to talk to other custom devices on I2C.  The canned I2C drivers from Atmel's ASF libraries get completely clobbered by the PixHawk code.  When switching periopherals, PixHawk does a bus check when it rechecks the bus speed (it allows each device to use a different speed).  This bus check was being flagged as a bus error by Atmel's ASF.  Didn't help that Atmel's code didn't handle the bus error gracefully.  The Atmel code would also lockup when this was encountered.  Although in that instance, the Atmel code would permanently clock extend and prevent all I2C traffic until a chip reset.

Just saying that PixHawk may be working fine until it hits a race condition in the LIDAR-Lite code which causes its drivers to hang.

On Monday, January 12, 2015 at 9:13:17 AM UTC-5, dcwalmsley wrote:

Need some help.


Last week I proceeded to bench test the LIDAR unit I just received and came across a peculiar issue.  

On several bench test I discovered that the LIDAR unit stopped sending data to the Pixhawk.  First attempt showed it stopped around 5.5 minutes, second test about 1 minute, third test was around 8 minutes, etc...
Further test showed it wasn't the Pixhawk which was the issue as resetting the Pixhawk didn't show it receiving data from the unit.  So I left the Flight Controller (FC) connected and monitored data and pulled power plug from LIDAR unit and then reattached it and the FC started seeing altitude data again.  This was done several times with same result.

So at this point, I think there is a communication issue between LIDAR Lite and Pixhawk.

I have left my concerns with Pulse Light on their blog in an attempt to find a solution to this problem.  See. https://groups.google.com/forum/#!topic/pulsedlight3d/JOjvyywFyWg

Please take a look and tell me if I'm not doing something right.   I will post my configuration settings this evening and latest log as well.

Thanks,
Doug Walmsley

[Dennis Corey]

All of the sensors with February 2015 or later manufacture date labels have the most recent updates to hardware and FPGA architecture including those meant specifically to address the issues associated with using LIDAR-Lite with a Pixhawk.  In all, these were significant (and costly) changes.  Needless to say, we are concerned and a bit puzzled as to why there are still reports of problems when we have worked hard to make our sensor compatible with the Pixhawk system.

There is a known issue with Cortex M3/M4 processors where a start immediately followed by a stop can cause the I2C peripheral to lock up, just as we saw with the "glitch" condition that existed in early versions of LIDAR-Lite.   The manufacturer has proposed workarounds, http://www.st.com/st-web-ui/static/active/jp/resource/technical/document/errata_sheet/CD00260217.pdf (Section 2.10.3) that, to our knowledge, have not been implemented in the Pixhawk.  There may be good reasons for not adopting these solutions, but noise on the I2C lines can be interpreted as this same type of event and the operating environment for Pixhawk is inherently noisy.  So even with the fixes in place in LIDAR-Lite, including filter capacitors that should help reduce noise on the data lines for all devices on the bus, I2C failures may still occur related to this specific problem.

Your experience and comments below also seem to suggest that using I2C for anything but the simplest devices may be problematic with the Pixhawk.  While LIDAR-Lite does not use an Atmel processor or drivers, our FPGA design has an I2C interface which was spec'd and built to "industry standards" but it seems that still may not guarantee full Pixhawk compatibility.  

Based on this information, our experience and known issues surrounding the Cortex M3/M4 processors related to the I2C interface, I would support your recommendation to use the PWM interface with Pixhawk instead of I2C.  That also appears to be the favored solution at this time of the Pixhawk developers.

[Tom Pittenger]

Dennis,

"We have pull ups because not all devices that can be run as master have them and we try to be as compatible with as many devices as possible."

Out of curiosity, which (type of?) devices communicate with the LIDAR lite that require you to use pull ups?

--

[Dennis Corey]

Many micro-controllers are set up with I/O ports that can be configured for a variety of different communications interfaces and they don't have built in pull up resistors because not all I/O interfaces require them.  We have also seen instances, such as the Arduino, where the recommendation is made to disable the internal pull up and use an external option and having it on our board makes the set up process easier.  There are also OEM applications where our sensor is used, where we have been asked to include the pull ups on our board.   

[Philip Rowse]

On a positive note, I have been having great results flying with the pulsed light Lidar on PWM mode on pixhawk....

Loving the sensor.

Phil 

[Dennis Corey]

Thank you Phil!  It's nice to hear some positive news! 

Dennis

[Tom Pittenger]

Same here! Other than that one start up issue, the old and the new LIDAR Lites have been working great for our needs. I commend you and your team for pushing the envelope in designing an impressive product for such a great price. There is, no doubt, issues when you run with the bleeding edge of tech.

[Dennis Corey]

Thanks!

Dennis

[Craig Elder]

Dennis,

>>>Many micro-controllers are set up with I/O ports that can be configured for a variety of different communications interfaces and they don't have built in pull up resistors because not all I/O interfaces require them. 

Please see section 3.1.1 on the I2C specification for the requirements for pull-ups and the use of multiple power supplies.

The appropriate solution is to add pull-ups to the transmission lines, not to the device.

[Dennis Corey]

The 4.7k ohm pull ups in parallel with the 1k ohm pull ups in the Pixhawk have a negligible impact on the impedance and work without issue in every other platform we've tested.  They are not the source of the I2C issues we are seeing with Pixhawk now.  

Even with all the changes that were implemented at the suggestion of the dev team to improve compatibility, I2C in the Pixhawk seems to be inherently problematic.

Until we hear otherwise from the dev team, PWM will be the interface recommendation for this platform regardless of sensor version.

[Lorenz Meier]

Dennis,

We have been enormously patient with Pulsed Light. We, and mostly / in particular Tridge, have spent enormous amount of time trying to help you avoiding a recall of your first thousand or so units, which we have proven are completely locking up in I2C mode beyond the influence of any external component. The I2C interface on them is essentially unusable, and it has been announced as one of the core features of the product at the time people ordered them.

However, given you continue to suggest the issues we have uncovered are not with your product without providing any data to help us understand that position, my patience has been consumed and I think we might be better off to publicly document our findings in the future here, so that your users and customers have the hard data we're basing our analysis of available as well. Transparency is key in our development community, in particular since it helps with safety.

Even with all the changes that were implemented at the suggestion of the dev team to improve compatibility, I2C in the Pixhawk seems to be inherently problematic.

Could you please quantify this comment? With quantify I mean actual data like logic analyzer traces or other useful data output which not only present the issue, but also, since you draw the conclusion, clearly show that the host is responsible for the issue. If it is we're more than happy to address it and it would be a very welcome and important contribution.

If I recall correctly someone from the DroneCode dev team has taught your engineer how to use a Logic Analyzer, so I'm sure you have no issues providing this.

I also assume that future revisions of the sensor will adhere to the I2C spec and not add device-side pull-ups to the bus? If each device vendor we are using would be following your logic of a "negligible impact" the impedance on the bus would be quickly unbearable.

-Lorenz

[David Pawlak]

+1 Lorenz

[Dennis Corey]

Lorenz,

Thank you for your comments.  

Patience is a two way street...

First, as a point of clarification, the I2C problem with early sensors did not make them "unusable".  Like all new technologies, they were not without their bugs and we corrected those as we found them.  The early sensors worked, and still work reliably for many people and for those that had problems we did our best to make them happy.  The implication that we intentionally misrepresented and shipped defective product borders on defamatory. 

We have supplied a number of the most recent versions of LIDAR-Lite to members of the dev team over 3 weeks ago with the understanding that the sensor would be tested by them to verify that all of the changes we had made to specifically improve Pixhawk compatibility were working.  Other than a bench test in Berkeley by Tridge and Craig, at which time they informed us that the initial results looked good, there has been no additional feedback from anyone on the team.   

I say the the I2C communications issue with the Pixhawk is problematic, because in spite of all of our work, there are still a few customers who are having problems with I2C on this specific platform.  We have been working hard to understand the issues, asked for assistance from the dev team, provided the team with customer data and we are all waiting for assistance - not lectures on pull up resistor specifications.

While we appreciate all of the work and support that people like Tridge have provided in the past, we are still a startup with limited resources who has been extremely responsive to resolving issues within our area of expertise as they have been presented to us.  This has not been without significant cost to us, but we felt that it was the right thing to do.  However, we can not continue to work in a vacuum in relation to the Pixhawk issues.  You are the experts in that area.  Has anyone in the dev team, who has been provided the new versions of the sensors, actually done any further testing as we have repeatedly requested?

I would suggest that we go off-line and schedule a call to discuss these issues, develop a plan for their resolution and then return to this forum with answers.  Transparency and discussion are fine, but they don't necessarily lead to solutions and often turn into a useless exercise in finger pointing.  Let's focus on working collaboratively toward solutions.  We owe it to all of our respective customers and to the community.

Dennis

[Lorenz Meier]

Hi Dennis,

Thanks for your feedback. I do perfectly understand that weeding out the very last few issues can be very time consuming, in particular on I2C and if multiple components or setup specific aspects are involved. We have spent enormous resources in the past on finding I2C related issues, in particular when users had long cable runs and / or many sensors on the bus. The additional noise on the bus triggered previously unhandled error conditions.

One of the challenges with an ivory tower style development model or a single point of contact is that its very hard to keep the right people in the loop, which is why I love mailing lists. Despite being the maintainer for the PX4 architecture I have not heard of any I2C issues for the last month, nor did I receive one of your updated units for testing (or anyone else of the PX4 dev team in Switzerland, as far as I’m aware). If you would like to revise that strategy we’d be happy to validate one of your new units in Zurich.

Is there an opportunity for you to get one of the problematic (partial or complete) setups off your customers? There is nothing compared to debugging with the actual hardware, and we have shipped customer setups to 3DR quite a bit in the past to recreate issues. Since I don’t know anything about the nature of the issues you’re referring to I can’t advise anything specific to try.

Hoping we can resolve this together, as we would like to use the sensor in our research at ETH as well and have three of the Kickstarter generation units here.

Cheers,

Lorenz

--
You received this message because you are subscribed to a topic in the Google Groups "drones-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/drones-discuss/yBVeykPoZBk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to drones-discus...@googlegroups.com.

[Dennis Corey]

Hi Lorenz,

Thank you.  I'm sure that working together we can solve the problems quickly.

More units are on their way to you.

I'll see if I anyone wants to volunteer to send a Pixhawk set up for troubleshooting.  Do they send it to you or 3DR?

I will also forward the log files we've received from customers to you.  You may also gain some insight into the nature of the problem from this thread.  In general, it appears to be related to having multiple devices on the bus and the interaction of Pixhawk with those devices, especially if they are operating at different bus speeds.  I'll provide additional information along with the files.  Craig and Tridge have the same log files as well as new sensors so they may be able to assist you in the investigations as well.

Dennis

[Lorenz Meier]

Hi Dennis,

The different bus speed technique has been decommissioned months ago, so they must be using relatively old software or the comment was based on outdated knowledge of the codebase. I found the comment about the Atmel device lockup interesting and I’d like to follow up on it, however, the field we operate in needs to be noise tolerant, and in general not handling error conditions properly is just a hint to an immature design. To be fair this whole field is abusing I2C like little other applications do and I’m looking forward to move on to CAN.

In our experience any sort of error can happen on the I2C bus (and we’ve seen quite special ones on long cable runs in the past). So fundamentally we and Atmel and anybody else needs to handle bus errors gracefully. Its like if we would say we don’t support flying in wind, because the disturbance to the control system is not supported 8).

-Lorenz

[Simon Wilks]

Hi Dennis, Lorenz

No one really knows this but I silently ordered two new Lidar units from Dennis and received them a few days ago so someone on the PX4 dev team does have them now :) I won't have a chance to test them for the next days though but could bring one over to ETH (I'll be there on Thursday...something I've kept secret too;) if that will be helpful.

Simon

You received this message because you are subscribed to the Google Groups "drones-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.

[Dennis Corey]

See!  We're already making great progress!  

We learned about the Atmel lockup from an OEM customer a couple of months ago.  There are workarounds, proposed by Atmel, that our customer implemented in their system which made it very noise tolerant and reliable.  For anyone using those processors it's almost unavoidable that this is a vulnerability that would need to be addressed and I'm glad you're looking at it!  I assume from your comments that Pixhawk already has ways of dealing with bus noise, but this is certainly interesting as it is at the processor level.

We don't have that vulnerability since we don't use that micro, but that is not to say there isn't some other interaction happening.

Units are on the way.

Dennis

[Dennis Corey]

Hi Simon,

That's great news too!  

I did know about the order, but didn't know you would be visiting Lorenz.  Awesome.  We do have a couple units on the way to Lorenz, but you arrive before they will.  Even a little time spent working with one of yours may tell us all a lot quickly.

Dennis

[Simon Wilks]

I'll hook an old one up now to make sure it fails then switch to the new one and take it from there. I'll report back here once I have some news.

[Dennis Corey]

Awesome...

[Simon Wilks]

The old Lidar failed within 10 min. when left with the lenses blocked (zero distance) and gave bogus readings before freezing. I hooked up the new Lidar and left it running overnight in the same position and it locked up at some point staying at zero.

I re-tested the new Lidar using Tridge's updated driver and it failed within 10 min (stuck at zero).

I used a Pixhawk v2.2 powered via USB only so I hope that rules out any power noise issues. So it it sounds like sticking to PWM is still the only reliable option for now.

Simon

[Simon Wilks]

One more thing, the Lidar was the only device on I2C. Other than that only a receiver was connected to the Pixhawk, nothing else at all.

[Dennis Corey]

Hey Simon,

Some interesting information and PWM is still our recommendation until this issue can be tracked down further.

The only problem with using USB to power everything as we've discovered and been told by a couple of people at 3DR is that the Pixhawk along with the sensor (100ma in operation) can push the limits of the available current on the USB port. So it might be a good idea to use a powered hub or separate power source to make sure there's enough current available.

Thanks for sharing the test info!

Dennis

[Tom Pittenger]

I second that, USB power works for me if it's from my laptop but not through the hub on my monitor which means it's right on the edge of what USB will supply, or at least what it needs to start up with.

[Simon Wilks]

Ok thx. Makes sense.  I'll pick up a powered usb and retry tonight.

[Tom Coyle]

Hi Dennis,

The only sure way to power a full up Pixhawk and associated components is not with a powered USB hub that I have found to limit the current to 500ma or less according to spec.

I purchased an Adafruit USB breakout box and cut the power traces on the input side from the PC and attached a 5vdc 1.5 amp power supply to the output side of the USB breakout box and it works great.

Regards,

Tom C AVD

[Dennis Corey]

Thanks Tom!  That's a good bit of information.  Another thing we've done in applications where they sensor is being powered from a battery (such as a handheld rangefinder) is put a 680 microfarad electrolytic cap across the 5V power leads going to the sensor.  This add a measure of reserve energy when the sensor is powered on and an acquisition is taken.

Dennis

[Tom Pittenger]

When powering via ESC, is it helpful to add additional DC bypass caps on the power or is there sufficient capacity on the LIDAR lite PCB?

[Dennis Corey]

It's not necessary, but can't hurt...  Might be an interesting test to conduct.  I would imagine that on a drone you wouldn't see much difference due to the amount of current available.  In our battery powered devices a 3v cell is used and runs through a power supply circuit to boost power to 5v and also to 20v to run a higher power laser.  When everything is powered up there is enough momentary load that the battery can't supply enough current. Without the cap the system would shut down.

[Axel Weckschmied]

Hi Philip,

do you connected LIDAR via PWM following the information from this site: http://copter.ardupilot.com/wiki/common-optional-hardware/common-rangefinder-landingpage/common-rangefinder-lidarlite/  ?

I've done it but it doesn't work. I think there is still a little mistake at the table in line: J3  --> Ch7 Out Signal  ----> there is no Ch7, the picture below shows, that J3 is connected to Ch5

What is right?

Are some special parameter values for this issue?

Do you know something about why LIDAR not works in Auto-Mode?

--

You received this message because you are subscribed to the Google Groups "drones-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drones-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Axel Weckschmied]

Hi Dennis,

can you please help me connecting LIDAR-Lite to PWN correctly and getting it work?

I used this manual, bit it doesn't work: http://copter.ardupilot.com/wiki/common-optional-hardware/common-rangefinder-landingpage/common-rangefinder-lidarlite/

Using (2C works, but PWM nat :-(

Axel

Am Montag, 20. April 2015 06:20:54 UTC+2 schrieb Dennis Corey:

Thank you Phil!  It's nice to hear some positive news! 

Dennis

On Sunday, April 19, 2015 at 7:45:13 PM UTC-7, Philip Rowse wrote:

[Philip Rowse]

The drawing is correct

[Chris Anderson]

I've updated that page to correct a typo and add a bit more detail to the photograph. Can you recheck and see if your connections are correct?

Chris Anderson

CEO, 3D Robotics

ch...@3drobotics.com

[Dennis Corey]

Thanks for updating the wiki!  I'm sure that it's going to help immensely.  I only have one correction to the first paragraph.  The links to the data sheet and technical information are all in one knowledge-base now at http://kb.pulsedlight3d.com/support/home .  If that can be updated it will be perfect.

Dennis

[Dennis Corey]

Of course isn't that always the case...  The wiki page is now correct.  The area where I've had the most problems personally has been in simply getting the configuration right in APM Planner.  I'd make sure that you have the proper firmware and the correct sensor type.  Also make sure that the RNGFND_MAX_CM and RNGFND_MIN_CM are set properly.  If you have the power enable line connected, you may want to temporarily disconnect it to see if the sensor starts working in PWM mode.

[Chris Anderson]

Thanks. Updated!

[Dennis Corey]

Perfect!

[Simon Wilks]

Sorry for the delay. I had to wait till I could go and buy a decent powered USB hub. It supplies up to 2A and the Pixhawk/Lidar were the only devices drawing power. The result was the same though. After about 1 hour it locked up.

I saw someone on the Paparazzi mail list warning others about getting the Lidar Lite and cited another list where Teensy users were also having significant I2C problems. Casual searching shows a significant number of people having the same problem.

[Dennis Corey]

Thanks for the update Simon.  

This is really frustrating for us, because we follow standard I2C communication protocol and the peripheral used in our chip is in use in other products.  The piece of information that's missing is what's really locking up.  We can't replicate the problem and the only input we get is that something is locking up.  Can you tell if it is the sensor or the I2C peripheral of the processor?  Was anyone able to look at the interactions with a logic analyzer or scope to understand what condition was occurring at the time it locks up?  I was kind of hoping that Lorenz might have been able to shed some light on the specifics after your visit.

As a bit of commentary on the other micro-controller issues...  We had problems initially getting the sensor to work with the Arduino boards via I2C.  It would run and then the Arduino would lock up.  Resetting the micro-controller would get everything working again.  What we discovered was that the basic Wire library for the Arduino did not do a good job handling errors or the "nack" response that the sensor will provide if it is busy processing an acquisition when you try to do a read.  As soon as we used the DSS Circuits Wire library the problems went away.  We run sensors for days in testing using an Arduino Uno without a single lock up.  And the sensor version doesn't seem to matter.  The source of I2C issues in most cases with these devices can be attributed to how their I2C drivers handle errors or the "nack" response from the sensor.  It's nearly impossible for us to stay on top of all the forums and we are not experts on any of the micro-controllers that are available on the market.  However, when we are contacted directly by a user we are usually able to help them out with suggestions on how to work around these issues.  Maybe not with specific code or drivers, but with advice based on what we learned from the Arduino.

While we are still looking at this issue, it would be great if we could get a lead on what I2C communications conditions exist when the "lock up" happens.  Is it a "nack" from the sensor causing the problem?  Noise?  Power issues?  Address conflicts?  Any specific information would be helpful.