Root cause for failed UART interface on SBC

[davegravy]

I could use some help troubleshooting an SBC that will no longer communicate via some of its UART pins and what could be root cause, so I can prevent future failures.

I'm communicating with a cell module (Ublox SARA-R510S-01B) and communication was working reliably. I stepped away from the project for a couple months and I return to find it no longer responding. The module powers on normally but there is no response to AT commands unless I disable hardware flow control in minicom at which point there is corrupt response.

Replacing the SBC (same SD card, same ublox module) communication works fine - so it is likely the SBC.

The SBC is this:

https://conclusive.pl/static/docs/kstr-sama5d27/DS_020100_20220401_kstr-sama5d27.pdf

There's only two things on my radar that it could be:

1) One of the SBC's GPIO pins was exposed to voltage close to the absolute max.

The PWR_ON input of the cell module has an internal 10kohm pullup to its 4.08V supply. I had this connected directly to an output pin on the SBC which is set to use 1.8V logic. The SBC pulses this low to turn on/off the module. It doesn't say for outputs but the SBC datasheet lists 4.0V as the absolute max for inputs. My next revision will have an intermediary MOSFET but is it possible this overvoltage damaged UART I/O? The SBC output that drives PWR_ON still seems to work fine.     

2) The SBC was power cycled every few seconds for a couple days.

When I left it 2 months ago the 12V LiFePO4 battery died. The battery supplies a 5V buck converter which powers the SBC.  I plugged in my 15V DC power supply to a solar panel charge controller to charge the battery and the draw was too much for the DC supply so it "hiccup mode" reset itself periodically every few seconds over the course of a couple days before I realized. I think the SBC was powering on and off accordingly.

Is there any inspection of the SBC I can perform to narrow down a likely root cause? Is it worth setting up a logic analyzer/scope to get more detail?

[BennoG]

your Ublox is 1.8V uart and your SBC is probably 3.3V this can work but is running in the grey area.
So it can work and it can not work (even aging of components can switch from working to garbage)
Look with a scope to the signal levels of the old SBC.

Benno

[davegravy]

OK thanks, I'll check it out the UART lines with a scope.

The SBC has a jumper on it to set the VDD_IO to 3.3V or 1.8V. I've got it set to 1.8V, so both devices are expecting 1.8V. That should be ok, no?

[magic]

I'm not entirely sure, were both boards on a common PSU with a common ground?

I once blew a GPS module by powering it from a separate PSU. After ground connection momentarily broke, its I/O pins stopped working.

[langwadt]

I'm not entirely sure, were both boards on a common PSU with a common ground?

I once blew a GPS module by powering it from a separate PSU. After ground connection momentarily broke, its I/O pins stopped working.

the the supplies are not earthed that can put ~110V (low current) between them

[davegravy]

I'm not entirely sure, were both boards on a common PSU with a common ground?

I once blew a GPS module by powering it from a separate PSU. After ground connection momentarily broke, its I/O pins stopped working.

Common ground but there's a regulator between the supply side rails:

Power path is like this:

12V (battery) --> buck converter --> 5V --> SBC
                                                       |
                                                       |
                                                       --> linear regulator -> 4.08V -> cellular module

(Only because 5V is too much for the cellular module)

Decoupling caps on the buck converter and regulator, although hmm... I haven't checked the voltage ripple on either.

[davegravy]

On the faulty SBC I did a loopback test with no flow control, only RX<-->TX. Worked fine.

I then added CTS<--->RTS loopback connection and when I launched minicom and turned on hardware flow control the terminal stopped responding... SBC crashed. I rebooted and tried again, same thing - SBC crash.

Removed the physical loopback on CTS and RTS, but kept minicom configured for flow control, no crash (obviously no communication tho).

With connection between CTS and RTS removed, minicom configured for hardware flow control:

* Checked voltage before opening the port: 1.8V on both CTS and RTS
* Checked voltage after opening the port in minicom: RTS is 0V, CTS is 1.8V.

Connecting CTS to RTS, measuring current: 1.7mA

Tested all the same using a good working SBC with no crashing and CTS to RTS current of only 0.01mA.

What can I glean from the above?

Assuming all signalling on CTS and RTS when connected to the cell module is at 1.8V (I will confirm soon) what could have caused this?

[davegravy]

Is it best practice to put a buffer between the devices just to prevent any high current flows between them?

I don't have any kind of buffer or level shifter between the devices. I read that if the SBC is on before the module that current can flow through the interface pins and cause damage. Is this true even if the SBC hasn't opened the port and initiated communication?

From the SBC I can sense that the module is ON from it's INT_VCC pin, so in software I could prevent opening the port unless I detect the module is ON.

However the module does go to sleep for power saving, and the INT_VCC drops during this. I think I have need to keep the port open to listen for wakeup events from the module but I'm not sure.

[Nominal Animal]

Although I'm only a hobbyist, I play a lot with various SBCs (Odroids, La Frite, et cetera) and Linux-running appliances like routers and switches and so on.

Note that the linked PDF does not mention CTS and RTS pins.

Is it best practice to put a buffer between the devices just to prevent any high current flows between them?

Absolutely.  The UART pins go directly to the SoC, and as such, are terribly sensitive to ESD and overvoltage –– compared to say USB D+ and D- pins, which are surprisingly robust on most SoCs.

Especially with 1.8V logic, I recommend TI TXU0102 (RX + TX only), TXU0204 (RX+TX+CTS+RTS), and TXU0304 (SPI) level translators.  These have dual supplies, Schmitt trigger inputs, with each side providing their own supply at whatever signaling voltage level they prefer, and have served me very well.

I also keep jellybean 74LVC1T45 in SOT23-6 (like TI SN74LVC1T45DBV and Diodes Inc. 74LVC1T45W6-7), which have 0.95mm pitch, and can be dead-bugged AKA soldered directly to wires (with some protective heatshrink) without a board, that are also well suited for this, although you do need one per signal you use, and these do not have Schmitt trigger inputs (so intermediate-level inputs can cause high current draw).  These also have a direction pin, which I typically ground, so that A is output and B input.

I've also used isolators, since I like to use the original ACDC supplies instead of lab supplies while investigating (minimizing the number of changes in the system).  In particular, TI ISO6721 (RX+TX) and ISO6742 (RX+TX+CTS+RTS) are well suited for 1.8V signaling (and for 2.3V-5.5V signaling too), and not too expensive.

Both the voltage level translators and isolators benefit from 100nF supply bypass capacitor on each supply (between supply and ground).  The isolators also work as level translators, as each side has their own supply and thus logic levels; but as the grounds are separate, they can deal with the couple of hundred volts of 0V/GND level difference that is possible when powering from different supplies.

Mouser sells automotive TI TXU0204-Q1 in TSSOP-14 for about 1.5€ apiece, and TI ISO6742QDWRQ1 in SOIC-16 for about 2.3€ apiece in singles; and SN74LVC1T45DBVRG4 for about 0.5€ apiece.  You can also have some translator boards made at e.g. JLCPCB, which has TXU0204PWR (C4363888) for assembly for about 0.56€, ISO6721BDR for about 0.75€ (you'll need two, one for RX+TX, the other for CTS+RTS); or get those from LCSC and solder yourself.

I recommend AGAINST using any kind of bidirectional translators, especially at 1.8V, because they tend to have a small step in the low state when erroneously changing direction, and that step may be large enough (at 1.8V and similar low logic levels) to be detected as low-high transition in the SoC.  I haven't waded through the ST documentation on ATSAMA5D27 to see what the logic level thresholds are at 1.8V VCCio, though.
And since the abovementioned translators and isolators have served me well, I'm not interested in risking getting bit by the bidirectional translators again, even if they work well for someone else.

In your particular case, I'd definitely use a TXU0204PWR or TXU0204QPWRQ1.  It has Schmitt triggers on inputs, so even intermediate voltage levels (between logic low and logic high voltages given the supply) will not cause extra current to flow.  When either side VCC is below 0.1V, the outputs on both side go high impedance.  All you need is that chip and maybe a couple of 100nF X7R bypass capacitors (one between VCC1 and GND, other between VCC2 and GND).  If I knew where the RTS and CTS pins on this SBC were (I know RX and TX are the two pins on the inside, next to the RTC battery), I would have shown a tiny board in EasyEDA that could sit on top of the end of the two-row pin header with the level translator, exposing the four comms pins.  TSSOP-14 is a bit fiddly to solder with an iron, having 0.65mm pitch (7 pins on each side), but if you have flux and maybe some braid to wick out any solder bridges, it is definitely doable even for a sausage-fingered hobbyist like me.  See the attached image what a simple breakout board for TXU0204 can look like.  It is single-sided, and the capacitor pads fit both 0805 and 0603 100nF capacitors.  I like putting 3mm mounting holes on my board –– I normally use nylon M2.5 or M3 standoffs and nylon screws for this, and the safe head area is marked with white silkscreen.

[davegravy]

Note that the linked PDF does not mention CTS and RTS pins.

I should have clarified - I'm using one of the flexcom interfaces that I've configured in device tree to be an extra UART. PB28-31 and PC0. I also should mention I built a cape at JLCPCB with a socket for the SBC to plug into, so I'll have to bodge this in for testing before I spin my next board revision.

All you need is that chip and maybe a couple of 100nF X7R bypass capacitors (one between VCC1 and GND, other between VCC2 and GND). 

To be clear, both sides are 1.8V in my arrangement, and the TXU0204PWR says it supports VCCA=VCCB. But I'm not sure how to distinguish each supply in connection to this IC. Should I take VDDIO from the SBC and V_INT from the module?

Also not sure what to do with OE, just tie to VDDIO from the SBC? Or is that bypassing some important protection it's providing and I need the SBC to control it via GPIO?

I have a hot air workstation so I'll probably just solder to one of these

https://www.adafruit.com/product/1210

and then dupont-wire it between SBC and module before I commit to it in the design.

Thanks!

[Nominal Animal]

To be clear, both sides are 1.8V in my arrangement, and the TXU0204PWR says it supports VCCA=VCCB. But I'm not sure how to distinguish each supply in connection to this IC. Should I take VDDIO from the SBC and V_INT from the module?

Also not sure what to do with OE, just tie to VDDIO from the SBC?

Assuming by VDDIO you refer to the SBC RX/TX/RTS/CTS pins, and by V_INT to the module 1.8V, then yes.  You can then connect OE to either one.

If V_INT is some higher voltage, but the module always uses 1.8V logic levels for RX/TX/RTS/CTS, then use VDDIO for VCCA and VCCB, and connect V_INT to OE.  I normally connect OE to VCCA or VCCB myself, whichever is easier.

Whenever VCCA or VCCB or OE are low, or any combination of them are low, TXU0204 is disabled and all outputs high impedance.  You only need to control OE if you want to turn the outputs high impedance for some reason during runtime, or if you need to ensure the outputs stay low until your SBC/microcontroller is running in proper mode.  If you use the same pins that emit serial console data during bootup, then I would control OE with a GPIO pin (one that defaults to an input, and I'd add a weak external pull-down resistor).  If your RX and TX pins don't toggle too often before your software runs, then I'd connect OE to VCCA or VCCB.

I have a hot air workstation so I'll probably just solder to one of these

Would be nice to add the 100nF between pin 1 and 7, and between pin 14 and 7, for added stability, but yeah.

[davegravy]

You only need to control OE if you want to turn the outputs high impedance for some reason during runtime, or if you need to ensure the outputs stay low until your SBC/microcontroller is running in proper mode.  If you use the same pins that emit serial console data during bootup, then I would control OE with a GPIO pin (one that defaults to an input, and I'd add a weak external pull-down resistor).  If your RX and TX pins don't toggle too often before your software runs, then I'd connect OE to VCCA or VCCB.

Hmm. I mentioned in an earlier post that I measure different voltages on the uart pins when the SBC is booted but the port isn't opened in minicom or my application, versus when it is opened (but idle). It's possible I'll need to power cycle the SBC while the cell module is still on. Does this mean I should have my application control the OE, enabling only once it's ready to start comms?

[davegravy]

Would be nice to add the 100nF between pin 1 and 7, and between pin 14 and 7, for added stability, but yeah.

True, I could still bodge thru-hole caps to the breakout pins. Not as good as SMD in tight but better than without and maybe good enough for testing?

[Nominal Animal]

Hmm. I mentioned in an earlier post that I measure different voltages on the uart pins when the SBC is booted but the port isn't opened in minicom or my application, versus when it is opened (but idle). It's possible I'll need to power cycle the SBC while the cell module is still on. Does this mean I should have my application control the OE, enabling only once it's ready to start comms?

If you have unused GPIO pins, I'd use one for OE just to be sure, since your use case is complex.  I'd check the SoC documentation, and pick a pin that defaults to an input, and use an external pull-down resistor for it, so it will stay disabled until you specifically enable it.  It won't do any harm, that's for sure.

Note:

When no process has the serial port open in Linux, RTS and DTR are deasserted (high).
When a process opens the serial port, the kernel automatically asserts RTS (low) and pulls TX high (as TX idles high).
(This assumes normal active-low polarity on the UART pins.)

TXU0204 doesn't mind inputs up to 5.5V, regardless of what the VCC is.  The 1.8V VCC sets low-to-high threshold voltage to somewhere around 1V, high-to-low threshold to around 0.6V, and output high voltage (to 1.7V if current draw is less than 0.1mA; to at least 1.3V if the current draw is 5mA).

True, I could still bodge thru-hole caps to the breakout pins. Not as good as SMD in tight but better than without and maybe good enough for testing?

Sure.  For short distances like on SBCs, with bulk capacitance on the 1.8V supply, you might not need the bypass capacitor at all.  It depends on how much current does the Ublox module inputs draw, what the baud rate is, how long the cables are, and so on.