Cheap solution for a fast SPI analyzer/sniffer?

[Novgorod]

Hi,

I need to monitor/record rather fast SPI bus activity at ~33MHz over extended time periods (tens of seconds) to a PC for analysis and I'm wondering what's the most efficient/cheapest/easiest method to do this. Of course that's a trivial task for a logic analyzer, but at these bus speeds it requires at least 200Msps sampling rate with a very big onboard memory or a USB 3 interface for real-time streaming to the PC, which is too costly. I know there are certain attempts to use cheap microcontrollers as logic analyzers, but they have too little onboard memory, so even if they're moderately fast (ESP32 or Pico?), the USB interface is a bottleneck.

Since I don't need a general-purpose logic analyzer, I thought there might be specific solutions for recording SPI as externally clocked 2-bit (MOSI and MISO) serial data and sending it as deserialized bytes over USB to the PC. In that case there's no need for oversampling because the data recording would be driven by the SPI clock and no overhead from "non-data" channels, so USB 2.0 should be plenty for the maximum theoretical data rate (66Mbps). However, I couldn't find any such "passive" SPI monitor solutions, only USB->SPI master bridges like the FT232H (I found an SPI slave project for the Teensy, which might also work in principle as a pure sniffer, but the dev said it can't go faster then 22MHz). The FT232H can also operate as a general-purpose FIFO (i.e. bit-banger), and it also supports asynchronous mode, i.e. being driven by an external clock. However, the maximum external clock speed in this mode is 16.7MHz according to the data sheet and also the USB transfer speed is far too slow (both are much faster when the FT232H is used as the clock source in synchronous mode, but I need to lock onto the external signal).

If there's no dedicated SPI monitor available for this kind of speed, then probably the asynchronous FIFO is the way to go, though it's kind of wasteful because you have to capture/transfer at least a whole 8-bit register per clock without any on-board processing. That means 33MBytes/s, which is very close to the USB 2.0 limit, but maybe still possible. Does anyone know a suitable (and cheap) interface for that? I've looked a bit into the Cypress FX-3 dev board, which is just affordable enough and has "unlimited" (USB 3) transfer speed for this matter, but I'm not sure whether it can implement an asynchronous FIFO to sample the SPI bits exactly. I know it can be used as a "conventional" logic analyzer (with its own clock) and allows continuous streaming with (at least) 8 bits at 100Msps, but that's just barely too slow to reliably record 33MHz SPI without clock sync...

[ataradov]

FX2LP, which is used in all the cheap logic analyzers can use external interface clock up to 48 MHz. It will also have no issue passing that 33 MHz stream to the host via HS USB.

But you would need to make custom firmware and software. I'm not aware of anything standard that would work. You may get away with modifying one of the open firmware versions and then use Sigrok or the Saleae software .

[Nominal Animal]

I'd use a Teensy 4.0, and two of the SPI peripherals in slave mode, one for each of the data lines, clocked by the SPI SCK line.  I've verified it can sustain 25 Mbytes/sec (200 Mbit/s) over USB Serial, so it would be a quick thing to use Arduino+Teensyduino to create a sketch that reads those into buffers using DMA, with the main loop writing each buffer when full or timeout elapsed to the computer via USB Serial.
I don't know the maximum SPI slave clock frequency supported, but 33 MHz master SPI clock should not be a problem.

Sampling just the SCK and MOSI pins at a fixed frequency is a bit more work.  You use a separate SPI peripheral for each signal line you want to sample, and use DMA to read them into a buffer.  You connect their SCK lines together, and either make one master and the other slaves, or connect all of them to a PWM pin.  (FlexPWM maxes out at 37.5MHz when running at 600 MHz, 33 MHz when running at 528 MHz; the SPI clock can go a bit higher but has fewer possible frequencies.)  Each will use a DMA channel to read data into say a 504-byte buffer (so you have eight bytes to identify the channel and buffer, and still fit in a single USB 2.0 HS packet); if you use a SPI clock, then you may need one more DMA channel to write the same byte/word (doesn't matter what) to the SPI output to enable the clock.  When a buffer is full, you update the DMA for the next buffer (in an ISR), and tell your main loop this buffer is ready to be sent.

The main loop just waits for buffers to complete, checks for incoming connections and for the userspace application closing the device (Serial will evaluate to true only when an application has the device open, at least in Linux), and sends each buffer in turn via Serial.write(buffer, len); whenever Serial.availableForWrite() >= len.  Note that in both approaches above, the data is a sequence of bits in each packet, with different packets providing data for different channels.  So do expect to include a short header (one to eight bytes) identifying where the bits are coming from.

For three sampled signals, I expect 60 Mbit/s each to be feasible; for two, 90 Mbit/s each.  (This is sustained indefinitely, as long as the receiver does not stall too much.  I recommend against using Raspberry Pi's for this, as their USB hardware is wonky, and can drop packets without informing anything.  Most other Linux SBCs (with either USB3, native SATA, or native PCIe interface) should be able to sustain this indefinitely to an SSD, using either PCIe, SATA, or a cheap USB3-to-SATA adapter like the JMicron ones.  If you use Dual or Triple Serial, you can squeeze a bit more bandwidth, but then it is very important each USB packet has a sequence number as they do not necessarily arrive at the userspace program in the order they were sent, being separate "devices"; otherwise you cannot sync the different bit sequences.  Dual/Triple Serial is especially useful if your machine is otherwise relatively idle, and you have two or more CPU cores.  I do suggest using a C program with a separate thread for each USB Serial port.

For FX2LP, I do recommend taking a look at Sigrok and its fx2lafv firmware.  Alas, the existing cheap ones (<$10) use a 24 MHz clock.

[ataradov]

Existing FX2LP devices would have to be modified anyway, since that 24 MHz is internal. You would need to bring out the external pin and then you can feed any clock you like. From the firmware point of view, it is just one bit setting to switch to the external clock.  And I think existing software would just work for the capture.

One issue would be synchronization of the bytes. So, you would need to make sure there is a reasonable way to determine the bit shift in the stream.

To me this sounds like the easiest thing to do.

[SiliconWizard]

If you need a "sniffer", then a single-channel SPI bridge will not work, so the FT232H is off the list, even if it could handle the frequency. You can configure a single SPI device only as slave or master, but can't "read" two bits at the same time.
For that you'd need to use 2 SPI channels, both configured as slave, both using the same clock signal, both only using their MOSI signal to MOSI and MISO to be sniffed. The FT2232H could do it, I think, as it has 2 channels. But I seem to remember the max freq. is only 30 MHz. (I don't know for sure where you got this 22 MHz figure from, but still you'd be over its limits here. Maybe it's 30 MHz as master and much less as slave, which would be understandable.)

If you think of using a non-specialized, cheap logic analyzer instead, you'll be in for some disappointment IMO. I don't know of anything "cheap" that will handle properly logic @33 MHz. It doesn't sound like very high, but it already is for cheap analyzers. Though, we don't know your budget.

I'm afraid there isn't really anything off-the-shelf that will be fast enough, cheap and not require development.
You could do this with any MCU that supports USB HS, but will have to implement it yourself. The FX2LP mentioned above may work (although i'm not sure you can properly sample SPI @33 MHz with it), but unless you can find existing firmware, have in mind that these are 8051 cores that require some learning curve to use efficiently. I've used the SDCC compiler successfully with these in the past, but it does take some experience, certainly not a picnic compared to using say an ARM Cortex.

The CH32V307 that we've been talking about in other threads would perfectly do it, with a much more capable MCU and modern tools. Still significant development time ahead.

Thing is, if you are looking for something off-the-shelf, while 33 MHz isn't that high, it's already at the limit of what cheap tools can handle usually, and after that, you'll need to resort to much more expensive development tools.

[ataradov]

You don't need to start from scratch with FX2LP. All you need to do is take Sigrok firmware mentioned above and modify one bit. It does not need to be efficient, since the core does nothing once FIFO is configured. And it can easily sample at 33 MHz. This can be realistically done in a day even if you never touched FX2LP before.

[SiliconWizard]

You don't need to start from scratch with FX2LP. All you need to do is take Sigrok firmware mentioned above and modify one bit. It does not need to be efficient, since the core does nothing once FIFO is configured. And it can easily sample at 33 MHz. This can be realistically done in a day even if you never touched FX2LP before.

Sampling at 33 MHz can't allow you to decode a SPI stream clocked at 33 MHz. Unless it's able to use the SPI clock as an external clock for the sampling, and work at this frequency? I don't know if it can. Maybe you do.

[DavidAlfa]

FX2LP, which is used in all the cheap logic analyzers can use external interface clock up to 48 MHz.

Nah, 48MHz doesn't work!
I've tried it in the past, it's just able to capture few milliseconds - maybe 100ms in a good day - before stopping (As stated here).
It can't work as in 24MHz mode, where it can capture data continuously.

Also 33MHz signal vs 48MHz sampling might cause some nyquist issues.

[Novgorod]

Thanks a lot for the answers, that's already very helpful. Right, I would need 2 SPI slave modules to capture both MOSI and MISO of the device under test, and then somehow package up both data streams in a synchronized way. The FT2232H (like the FT232H) can't be used as a slave apart from being too slow. The Teensy has multiple SPIs with DMA but it seems difficult (or not intended) to be used in slave mode - that's where I've got the 22MHz from. And that seems to be for a single SPI slave channel, so the bottleneck there is not the USB transfer speed but reading the SPI bus as a slave.

So I guess asynchronous FIFO should be the right way? The "overclocked" FX2LP sounds interesting, but does it really work in asynchronous mode, i.e. with a variable external sample clock? The FTDI competitor of this chip (FT232H) has substantial speed limitations in this mode. The sigrok implementation seems to treat it like a normal logic analyzer with sampling at some constant clock rate. However, for this application I want to use specifically the external device's SPI clock as the sampling clock to avoid "Nyquist issues" and to be able to use the same sampling rate as the actual bit rate - but this clock is not constant (bursts of 8 clock pulses at 33MHz with pauses in between), so even if I can tell the FX2LP to use an external clock, I assume it requires the external clock to have a known and more or less constant frequency because everything else (CPU, USB) relies on it, right? Therefore I don't think it would work with a stuttering SPI clock as the "system" clock, that's what asynchronous sampling is for. I don't know how this is implemented though - if it's CPU-based GPIO with wait loops or interrupts, then the sampling speed would be limited to some fraction of the internal clock; if it's some hardware-triggered buffer, it can be precise at any speed as long as the buffer is read out fast enough before the next sample.

Regarding the budget, I'd say up to the price of a Cypress FX3 board (~50 bucks) is reasonable. A suitable logic analyzer (3 bits, 200Msps, 10+ seconds recording time with compression) would cost at least $200 or so. I'm just wondering whether the FX3 (being much faster than the FX2) could be used as an asynchronous FIFO at these speeds...

[Jope]

You could use an FPGA board with enough RAM, like the Digilent Arty or cheap alternatives from China, like the ones from Qmtech, e.g.: Artix-7 XC7A35T DDR3 Core Board, which has 256MB of DDR RAM.
If you've never done anything with FPGAs though, this will be quite a task.

[DavidAlfa]

Or a CYUSB3KIT-003 and this firmware:
https://community.infineon.com/t5/Knowledge-Base-Articles/EZ-USB-FX3-Explorer-kit-as-16-channel-100-MHz-logic-analyzer-with-sigrok/ta-p/283993

[ataradov]

I've tried it in the past, it's just able to capture few milliseconds - maybe 100ms in a good day - before stopping (As stated here).
It can't work as in 24MHz mode, where it can capture data continuously.

  May be Sigrok software sucks at receiving the data. FX2LP itself will have no issues with 33 MB/s capture. So, you will also have to use custom capture software. I think I've seen generic examples for that too. Or may be Saleae software is better.

Also 33MHz signal vs 48MHz sampling might cause some nyquist issues.

Unless it's able to use the SPI clock as an external clock for the sampling, and work at this frequency?

This is exactly what I'm proposing.

[S. Petrukhin]

I did not understand why a high sampling rate of 200 MHz is needed.

If you want to observe the data stream and not the waveform, then 33/8=~4 MB/second.
The MCU will receive a serial signal and output it decoded as byte stream - any MC is capable of this.

Modern inexpensive ARM have from 64 to 1024MB of RAM, i.e. from 16 to 128 seconds of data stream you can capture in RAM and then transfer to a PC slowly.

Even an Arduino with five lines of code is capable of this, preferably a bluepill with stm32.

[S. Petrukhin]

And it is absolutely necessary to use the  master CLK in order not to record silence.
Preferably CS, because it is often used as a signal for the beginning/end of a packet, for example, the W5500 works like this.

[ataradov]

If you want to observe the data stream and not the waveform, then 33/8=~4 MB/second.
The MCU will receive a serial signal and output it decoded as byte stream - any MC is capable of this.

It will be twice that because you need to capture both MISO and MOSI.

Modern inexpensive ARM have from 64 to 1024MB of RAM, i.e. from 16 to 128 seconds of data stream you can capture in RAM and then transfer to a PC slowly.

What inexpensive ARM has 1 GB or RAM? It would be SoCs, which makes access to two SPIs and capture at 8 MB/s questionable.

Even an Arduino with five lines of code is capable of this, preferably a bluepill with stm32.

Where are you going to store the data in Arduino?

[S. Petrukhin]

If you want to observe the data stream and not the waveform, then 33/8=~4 MB/second.
The MCU will receive a serial signal and output it decoded as byte stream - any MC is capable of this.

It will be twice that because you need to capture both MISO and MOSI.

Modern inexpensive ARM have from 64 to 1024MB of RAM, i.e. from 16 to 128 seconds of data stream you can capture in RAM and then transfer to a PC slowly.

What inexpensive ARM has 1 GB or RAM? It would be SoCs, which makes access to two SPIs and capture at 8 MB/s questionable.

Even an Arduino with five lines of code is capable of this, preferably a bluepill with stm32.

Where are you going to store the data in Arduino?

I agree, both directions will require 2 times more memory.

STM32F407ZG = $5

• CLK SPI and SPI2 are connected together to CLK master
• MISO SPI1 to MISO
• MISO SPI2 yo MOSI
• turn on DMA and add it to memory at a speed of 168/2/8*2=20Mb per second.

The bluepill has 64Mb of RAM - a few seconds will fit. 

I have many devices that work steadily and without problems with the W5500 on SysCLK/2=84MHz

[sicco]

On a Teensy 4.1, here’s how i did it: https://forum.pjrc.com/index.php?threads/spi-slave-for-t4-dma-spi-slave.73418/

So two hardware SPI ports, both in slave mode, that capture MISO and MOSI data in DMA buffers.

[ataradov]

What "bluepill" you are talking about? Usually by "bluepill" people understand a board based on STM32F103, which has 20 KB or SRAM.

[DavidAlfa]

Or may be Saleae software is better.

AFAIK, Saleae version is 24MHz-only.

[S. Petrukhin]

By the way, you can leave USB alone and use faster Ethernet.

With the w5500, it is not at all difficult to organize a data stream even using the STM32F103ю.
You can use 2 sockets with their built-in 8MB memory in pinpong mode - one socket fills up, the other transmits.

[iMo]

Rpi Zero 2w has got 2 SPI you may utilize. Will do 50+MHz SPI clock sure, 512MB of dram, 4 core 1GHz cpu, and XXX GB of flash on the sdcard (it can write 10MB/sec sure).. $15 they say..
And there are tons of others for similar $$ floating around..

[S. Petrukhin]

What "bluepill" you are talking about? Usually by "bluepill" people understand a board based on STM32F103, which has 20 KB or SRAM.

No, I'm completely broken... I confused SRAM and flash. 

[ataradov]

AFAIK, Saleae version is 24MHz-only.

It can set 24 MHz sampling rate, but you don't care about that if you are using external clock. As long as it can receive 33 MB stream, it will be fine. You are not getting protocol decode anyway, since there will be no clock. All you get is export in CSV or similar.

But Seleae is probably not going to work, since I assume it will want to overwrite the firmware each time. So, it would be hard to substitute the hacked firmware. And the firmware commands would have to match Seleae, so this is entirely out.

Anyway, a custom tool that dumps raw data is not that hard to make. My USB sniffer does this in the speed test mode. It just does not save the stream to a file. But that is trivial to add.

But I'm still sure sigrok can receive 33 MB stream. Same story, you don't need to set 33 in the UI. Set whatever, as long as firmware itself pushes the data at 33 MB/s, sigrok will not know the difference.  The only bottleneck would be that it is too slow to handle that much data. But for me it works fine at 24 MB/s, so I don't really see it breaking at 33.

[DavidAlfa]

Hacked firmware? FX2LP boards have no ROM, no firmware, it's loaded every time when plugged in and starting the program.
The only thing identifying it is the USB VID/PID programmed in the external eeprom.

If you're talking about uploading your own fw then using Saleae, not sure about that, but I guess it won't upload it again if it detects it's already running?
If so, does it make sense to you? Maybe you're used at developing firmware for it, but 99% of people won't. We're talking about something simple and straighforward.
We started talking about analyzers, now it's about developing our own firmware and pc client software!

[ataradov]

Hacked firmware? FX2LP boards have no ROM, no firmware, it's loaded every time when plugged in and starting the program.

EEPROM can absolutely store the firmware. I don't know if the cheap logic analyzers include EEPROM big enough, but the commonly available cheap development boards do include  large EEPROMs.
 
But Saleae software itself would not work anyway, since it will need specific Saleae firmware to hack, and I don't know if there is open source code for one. It is really not worth it, as I'm sure sigrok will be able to receive 33 MB stream. You still won't get decoding or even byte boundaries. But it will be enough to export.