PIC24 SPI arbitration problem

[hans]

I've a project with a PIC24FJ64GB004 connected to an ENC28J60 Ethernet chip and a MRF49XA(or RFM12B/Si4421) RF transceiver. It's a RF<>ethernet gateway, basically, for personal domotica project. The problem is that I've connected both chips on the same SPI bus ( ), and I don't really want to hack the board just yet (I am not planning on re-spinning this board soon, unless I run out of memory some day).

I've got both ethernet & RF stacks working fine on their own. The ethernet driver + stack serve (simple) webpages and static content from a FLASH chip (which is connected to SPI2). The RF stack can handle plenty of ping/pongs very well.

The issue is that RF comms runs from an ISR and Ethernet in the main loop (lower priority). That's because of the RF baudrate (21kbps/1 byte every 0.5ms) and, moreover, the "FIFO" on the RF transceiver is only 1 byte deep. So if the RF transceiver interrupts the PIC, it should read/write the byte ASAP.

Inevitably this ISR fires while the ethernet driver is talking to the ENC28J60 and transmitting or receiving an ethernet frame (which can be up to 1500 bytes). The SPI communication seizes obviously, because there are 2 chip selects active at once.

First thing I tried was to disable interrupts when the ethernet driver was using the SPI bus. This made ethernet run very well, but RF very badly. I get virtually a byte loss on every 20-byte RF ping/pong packet I transmit.
I simulate a small ethernet load with ethernet ping/pongs that are 98 bytes long. 98 bytes * 8 @ 4MHz SPI bus = 0.2ms. That is quite a significant amount of time to have ISRs turned off.

Second thing I tried was to have the ISR routine abort any ethernet SPI communication. I rewrote the low-level routines to retry the command if it was aborted, like so:

Code: [Select]

void enc28j60WriteUint8(UI08_t reg, UI08_t value)
{
    do
    {
        while (!spiArbEthAcquire());

        spiWrite(1, WCR | (reg & 0x1F));
        spiWrite(1, value);

        spiArbEthComplete();
    } while(spiArbEthWasAborted());
   
}The ISR routine would deselect the ethernet chip and flag the transfer as "abort". Acquire will check if the RF is busy and reset the abort flag. For data this would be fine I suppose (although a big time loss if it would abort at the last byte)..
however I believe for some registers this is a problem because their behaviour is like "bitset this field to decrement packet count by 1" - so I got weird packet loss/buffer problems.


Any other idea's how to solve such a problem? I can't help but imagine this will happen more often, but can't find so much info via Google about it.
Or perhaps I should get the Stanley blade out and re-route the ethernet chip to the second SPI bus?

[dannyf]

The obvious choice is to use separate spi modules for this.

If that cannot be done, i would use a buffer to save the data to be transmitted to the rf module if the spi module is busy - there is a flag for that. You only transmit to the rf module when the spi module is not busy.

[hans]

(Editted first post a bit, some double meanings removed)

I (already) maintain a packet buffer for 16 packets, both TX&RX. The ISR only moves the "raw" bytes around. On TX it also the transmit scheme(preamble, network ID's, etc.). I could probably fix something where the ethernet stack would be completely stalled when the RF is going to transmit a packet.

So the problem is mostly for RF receive, as that can happen at any time and needs to be read before the next byte is being received.

[macboy]

Try this:
Mask the RF interrupt while transmitting/receiving a few Ethernet bytes (actually while the Ethernet CS is asserted). After the Tx/Rx is complete (you'll need to poll the USART status), de-assert the CS, then unmask the RF interrupt, only for a single cycle. If it is waiting to fire, it will fire and the ISR will do its work before the next instruction executes, guaranteed. The next instruction will mask the interrupt again, assert Ethernet CS, send/receive few bytes, then repeat... de-assert Ethernet CS, unmask the interrupt, etc.

With your ENC28J60 Ethenet module, you can break up the reading/writing the Tx/Rx buffer memory into smaller chunks of a few bytes instead of the entire buffer at once. This will minimize the time that the CS is asserted (and RF interrupt is masked).

This should allow sharing the SPI while maintaining what is hopefully a low enough interrupt latency to allow your RF to work too.

[hans]

With option #1 I tried, I only would enable/disable interrupts before&after the moment Chip Select is touched (is done inside the EthAcquire() en EthComplete() functions). I've now changed the block transfers to a maximum of 16 bytes at a time, that's a good idea. I use the DISI 0x3FFF instruction to first disable interrupts, and DISI 0 to enable them again.

It's seems to be more stable now, however every time I reload a webpage it will still corrupt 1-3 RF packets. When I ping at 5Hz from my PC, I see 1 per 8-9 packets being lost too.
It occasionally also corrupts a packet with "background" traffic that I receive but ignore, for example I get a lot of traffic from my PC when the link just gets online, with requests that's asking for a gateway/router/DHCP/NTP/etc.

If I disconnect the ethernet cable all together, I don't see a single fault in half an hour.

I've taken some time measurements in the enc28j60 routines and I saw the processing of an ethernet frame for the webserver can take up to 7ms (rendering a table with TCP stats), but a ethernet ping/pong response takes 0.7ms to complete. 2x200us of that is dedicated to reading & writing the packet it self.
It always seems to trigger when I load a webpage (that doesn't block ISR or use SPI1), but as it also happens on just random traffic, it confuses me what could cause it

[SPRX]

Is it an option to modify the RF to run without Interrupts?

See if you could incorporate both apps to move through the state machine without using Interrupts, if your RF module permits.

[hans]

For those really interested in the code base, here it's on github: https://github.com/nlhans/domotica-rf

I decided on interrupts because of the shallow FIFO, the randomness of receiving packets and the low baud rate. If I'm right, the current baud rate setting means 1 byte every 0.5ms. The FIFO is 16-bits, or 2 bytes. So that means I would have to check the FIFO every 0.5ms - 1ms.
Because it will also serve webpages/HTTP at the same time, I need a way to interrupt that process. The datasheet doesn't mark the interrupt line as "optional".

So I don't think I have got much other choice.

I've been tinkering around a little bit more. I notice the more printfs() I do, the worse the problem (can) become. I use a very low-tech UART driver; I only set up the UART registers to 38.4k baud and let the stdlib take care of the rest. I read there were some major changes made on v1.21, but unfortunately on "legacy stdlib" the application crashes some where (address error).

I've also tried to look how much time the longest SPI operation takes. The block transfers got that down by a huge bit, however the timer says it still takes 110 - 120us to complete transferring 8 bytes.
I also noticed that the ping times are quite high (6ms), so I wonder what slows things down so much. Maybe it's the XC16 1.21 compiler again (I was getting 2ms on older compiler/simpler versions of this project).


I have tried to count the number of interrupts that are handled by software (in the ISR) and are sent out to the hardware (to timer 4 ext clk via remappable I/O).
The difference value (software - hardware count) dances at +/- 1, even if all is fine. Occasionally I see difference of 2 coming by, but it's not (always) when a packet is missed. So I'm not entirely convinced it's perfect.

[dannyf]

So that means I would have to check the FIFO every 0.5ms - 1ms.

No flag to indicate when the buffer is below certain levels?

Because it will also serve webpages/HTTP at the same time, I need a way to interrupt that process.

The solution depending on what you meant by "interrupt".

If I understood your issues correctly (which I doubt), you have a pack of data that you are transmitting and in the middle of it, you need to send this one byte.

You have to figure out which of the transmission is of higher priority; and if the lower priority transmission is interruptable.

Based on your answers to those questions, the solutions differ.

Generally, for transmitting large amount of data, the simplest would be to use an isr that loads the buffer when the buffer is less than full (ie it needs a flag that triggers the spi isr). A complex one would be to use a dma channel to drive the spi.

The dumber solution would be to poll the spi flag, and the dumbest solution would be to loop inside the isr.

[hans]

The MRF49XA signals an (external interrupt on the PIC24) interrupt when it's FIFO is filled up past a certain point. That is currently set at 1 byte, which is half of the FIFO size. At that point, it's interrupt routine will read the byte.

I have data coming (so the PIC24 is receiving) from another node. I view it as a random, spurious event, with no clear indication "now someone is going to send me something". The node may be waking up from sleep, read a sensor, send the packet, and go back to sleep.

That's why I don't want it to disrupt the ethernet process (because that may also be doing something), moreover also not the other way around like I see now: when there is (a decent amount of) ethernet activity the RF communication shouldn't fail. The failure type is that there is a packet recognized, but there is not enough data (so it must have lost a few bytes somewhere).

In my view RF has the highest priority, because after 0.5ms it will start losing data (FIFO overrun). The ethernet cihp can wait a while, because the ENC28J60 has 8kB worth of buffers, so it has plenty before it will run out. Moreover, I wouldn't necessarily care if an ethernet packet is lost.. TCP is robust enough to cope.

Now because both parts of the system will go their own way, there will inevitably a conflict at some time on SPI1.

What I've done now is disable interrupts when ethernet is busy in SPI1, but is turned on/off between calls. I've tried to minimize the time ISR's are off by reducing the size of long block transfers. But the stutters are still occurring. The external interrupt has the highest priority of all interrupts I've configured.

I don't see many other ways doing the SPI transfer. This part does not support DMA. I could use the enhanced buffer mode, which could shorten the delay between bytes; but the protocol to the ENC28J60 is semi-random; one register needs writing, the other needs reading, needs to change banks, etc.

However, it almost seems like the SPI problem is not the issue anymore, as I can see a similar effect happening when I , for example, do a large-ish printf in the RF section. For example printing the last read MRF49XA STSREG value.

[abyrvalg]

What do you prefer - nice board, but marginal timings in sw or a bit "patched" board, but reliable sw? I vote for cutting 4 tracks and soldering 4 wires to second SPI port. Finally you can put the board into some box and get both "nice" and "reliable"

[dannyf]

If I understood this correct (which i doubt): you have two senders of spi data, unpredictably, to the mcu where the mcu acts as a master. One sends you a small frame (1 byte) and another sends you a large frame (hundreds of bytes?), over the same spi bus. The small frame sender has priority but you do not want it to interrupt the low priority sender (what kind of "priority" is that?).

Good news and bad news.

Bad news: it cannot be done. It doesn't sound like you want to interrupt the low priority transmission when the high priority transmission comes in and it also seems that the low priority transmission is too long for you to be able to queue the high priority transmission.

You have to use two separate spi modules.

Here comes the good news: You can implement a bit-bang spi just for the high priority short frame transmission. For example, you can set up the large frame transmission via either dma (if your chip supports it) or interrupt (you essentially send a bunch of zeros and read the spi fifo buffer). mcu utilization would be close to zero (for the dma approach) or very small (for the interrupt approach).

The high priority transmission can be done in the extint or CN interrupt where you simply bit bang the input - simple and sweet.

Now, if you insist on putting them on the same bus, well, you just have to learn to live with missing transmissions: you can institute some form of a proprietary protocols (parity, crc, etc.) where you can detect errors and have the master request re-transmission.

[dannyf]

it almost seems like the SPI problem is not the issue anymore,

A lack of foresight is, .

That's why those people who have made big mistakes and learned from those mistakes are paid big bucks. Almost paradoxical.

[hans]

2x SPI: that's what I already was afraid of to begin with. I started this topic to hear about other insights.

Priority? Well, basically it's a priority based on a(my.. ) stupid HW design in the first place  that I try to software patch. The reason why I didn't want to cut & patch straight away, simply put, is because in my current situation I don't have access to the tools available to do that effectively. I have them. Just can't get them. A bit outside the scope of this topic.

That's why those people who have made big mistakes and learned from those mistakes are paid big bucks. Almost paradoxical.

Something along the lines of: people who don't want to be proven wrong will never come up with something new.

Can't remember where I got that phrase from.

It doesn't sound like you want to interrupt the low priority transmission when the high priority transmission comes in and it also seems that the low priority transmission is too long for you to be able to queue the high priority transmission.

Well, it should be possible I guess. I was already trying that but got distracted into the disable-interrupt route fairly quickly. That is because I would have to check which registers are sensitive to "write 1 to decrease X" or "resets on read". I could still retry that attempt, but it would probably lead to a (small) rewrite of the ENC28J60 driver.