ARM External Bus Controller Timing

[SimonR]

This question is a bit specialised but I wondered if anyone else has seen the same problem with the external bus controller on any ARM based microcontroller.

I have a current design that uses an Atmel SAM9 device, the at91Sam9XE to be exact, with three external peripherals connected to it.
One of the operations I do is to burst a block of data from one peripheral to the other one.
From memory this should take about 130nS, 80nS read and 70nS Write more or less.
Interestingly the time the code takes to run internally is completely hidden from this process so the read and write operations are truly back to back.

However the actual read write time is about 150 - 200nS longer and the reason for this, that the data sheet does not make obvious, is that whenever the bus controller changes direction it takes an extra 150-200nS. I cant remember the exact times but if I find the logic analyser traces I'll post them.

To confirm this I did 2 reads in a row and it takes 160nS, 4 reads takes 320nS. The same happens for writes, the timing is exactly what you would expect.

OK I thought there is another option. Lets read all the data into internal SRAM first as its got a 5nS access time and avoids changing bus direction more than once.
But that didn't work either. Even writing to internal RAM incurs the cost of a direction change.
I've reduced the impact a bit by doing 8 reads to 8 cpu registers and then 8 writes so there is now only get one direction change every 8 cycles, but its not really good enough.

I wondered if anyone else had noticed this problem on the processor they are using.
The reason I asked is that the next version of the board will get a new processor and I want to avoid this problem if I can.
I spoke to ST and they seemed to think their devices would do the same thing, although they weren't completely certain.
Reading the data sheet may not help either.

[jnz]

I'm not following the topic really... But are you running into the "speed" settings of the IOs? I don't remeber if it's STM32 or CMSIS that specs that each output has a "speed" of 2, 10, 50Mhz... I can't remeber if they were talking about switching speed or a programmable slew.

[exmadscientist]

It's hard to say without more detail, but I suspect you are running in to the details of the AHB bus matrix implementation inside the CPU. Atmel, of course, does not provide a bus matrix diagram for that part (that I could quickly find).

What is the data source, sink, size, and relevant clock rates (core, peripheral, and external if applicable)? That should shine some more light on what is reasonable.

I also have to mention that expecting hard realtime response from an MCU is sometimes not feasible. An FPGA might be a better fit for your needs. Newer Cortex-M parts also typically have enhanced internal buses but changing architectures has its own challenges, as always.

[rstofer]

Take a look at the block diagram and see where the GPIO pins are located.  If they are on the other side of an AHB contoller from the CPU, they won't run at CPU rate.  In the early years, the LPC2106 was a dandy ARM7 but people complained that they couldn't toggle the pins fast enough.  The LPC2148 came along and there was a Fast I/O bus that didn't cross through an AHB controller.  Instead the pins showed up in two places in the block diagram.  Same pins, just two different ways to get there.

You can download the relevant documents and see how this was done.

[Kilrah]

It's hard to say without more detail, but I suspect you are running in to the details of the AHB bus matrix implementation inside the CPU. Atmel, of course, does not provide a bus matrix diagram for that part (that I could quickly find).

+1.
ARM MCUs aren't like your good old 8-bitter where peripherals are directly in memory space and access is instant, they're complex beasts with multiple shared buses and interconnects with boundaries to cross - while your peripherals are still memory-mapped, they may reside on a different bus that has a different clock speed than your CPU requiring going through a bus controller, the bus you need to use to talk to them might be held busy for a while by other peripherals (e.g. during a DMA transfer) etc, so precise timing is hard to predict and actually quite variable.

A few months ago I came across a very interesting article from a guy who implemented complex 3D video routines on an STM32 and made a demo, and he went into great detail explaining the bus matrix and how to choose paths to minimize bus contention, but unfortunately I can't seem to find it anymore

[andersm]

If it is the bus matrix, according to the documentation it can be reprogrammed from the default round-robin configuration to a fixed-priority scheme.

[SimonR]

Thanks for the replies so far.
Some more information.

The CPU speed is I think 198Mhz, its what was set in the project I based this on, which just shy of 200Mhz.

My peripherals are NOT connected to GPIO lines they are properly memory mapped onto the external Address and Data busses.
This is an intermetiate microcontroller half way between SOC and pure microprocessor so it has all the usual onboard peripherals, 512K of Flash and 32K of RAM but it also has a proper external bus.
There are 8 decoded chip selects each accessing 256 Meg of address space.
The bus controller is fully programable and in theory you could programe it with a 5nS access time but in practice only 20nS or faster is practical.

Kilrah you are correct there are several clock domains to cross between the bus and the CPU.
The ARM core is conneted to a 6 layer Matrix (possibly 6 AHB busses, it doesn't say) and that is connected to the EBI (external bus interface).
The EBI contains an SDRAM controller, am SRAM controller and a NAND flash controller. My peripherals are connected to the SRAM controller.

Havinig said that if I set the timing to be 80nS access time then an ARM instruction should see 80nS. and it does see exactly that for sucessive reads or sucessive writes.

The external peripheral are
8Mbit SRAM 16 bits wide (10nS but not currently used )
Compact Flash 16 bits wide memory maped (uses the SRAM controller in CF mode)(fastest card is 80 ns)
An external comunications device 16 bits wide (80nS writes 70nS writes)

What I am trying to achieve is 10 Mbytes/S in a burst between peripheral and CF card, which is 5Mwords/S which is pushing the two devices close to the limit bus is entirely acheivable without the turn around issue.

I would consider using compact flash UDMA mode but a design cockup prevents this, so I am stuck with the basic bus timings.
I am considering an FPGA next time but other consideratins may get in the way of that idea.

[nctnico]

If the ARM controller has a DMA engine: have you tried using it to see if it works better/differently?

[SimonR]

Andersm thanks for the round-robin tip I hadn't spotted that, although I still think the EBI is the cause, its worthy of further investigation.

nctnico unfortunately DMA is a whole other problem that I had intended to post seperately.

This part doesn't have DMA that is of any use to me as there is no DMA available on the external bus, it doesn't even have memory to memory DMA internally or externally.
The only DMA is between onboard peripheals, and that is true of almost every modern device on the market.

I couldn't use memory to memory anyway because the comms peripheral uses good old fashioned REQ/ACK signals connected to an 18 word FIFO. Memory to Memory requires that you know in advance how much data to transfer, and I don't.

I had to implement DMA in software and that led to this problem. The software implementation is unbelievably quick considering it uses the Atmel GPIO library, easily as fast of faster than the 1980s hardware it replaced. The proble is that each REQ/ACK results in a READ/WRITE as described above.

Rumour has it that Freescale and I think NXP do actually have proper external DMA pins available but I have not yet invesigated.

[Kilrah]

I had to implement DMA in software and that led to this problem. The software implementation is unbelievably quick considering it uses the Atmel GPIO library, easily as fast of faster than the 1980s hardware it replaced. The proble is that each REQ/ACK results in a READ/WRITE as described above.

I might be talking crap, but couldn't you investigate "cheating" and duplicating some of the connections to GPIOs, and handle the REQ/ACK transations separately there in order to avoid changing direction on the main bus?

[SimonR]

I don't think you can cheat
If your thinking of handling REQ?ACK in a seperate thread (so to speak) from the READ/WRITE, then no because every REQ?ACK must have a read.
Then you have to store the value somewhere.

If you are thinking of doing a number of ACKs and only then doing a number of reads then no, the peripheral chip gets confused. I effectively tried this when debugging the DMA code.

If you are interested in how the DMA code actually works then here it is almost entirely. I had to merge the c file and the header but its pretty much complete.
If anyone else finds it useful the please feel free to use it

Code: [Select]

/**
 * \file        DMA.c
 * \brief       functions for the DMA engine
 */
/*
* Includes
*/
/* Standard headers */
#include <stdint.h>
#include <assert.h>

/* Library headers */
#include <board.h>
#include <pio/pio.h>

/* Dependencies */
#include "DMA.h"

/*
* Constants
*/
#define PIN_DMA_REQ     {1 << 13, AT91C_BASE_PIOC, AT91C_ID_PIOC, PIO_INPUT,      PIO_DEFAULT}
#define PIN_DMA_ACK     {1 << 1,  AT91C_BASE_PIOC, AT91C_ID_PIOC, PIO_OUTPUT_1,   PIO_DEFAULT}
#define DMA_MODE_IO2MEM (0x20)   /**< univerasl(I/O to memory) */
#define DMA_MODE_MEM2IO (0x40)   /**< univerasl(memory to I/O) */

/*
* Type definitions
*/
typedef struct DMA_s {
            uint8_t  Mode;       
            uint16_t *Memory;   
  volatile  uint16_t *Peripheral;
  volatile  uint8_t  *Peripheral8;
} DMA_t;

/*
* Private prototypes
*/
static void DMA_Request_ISR(void);
static void DMA_Process_8Bit(void);
static void DMA_Process_16Bit(void);

/*
* Public data
*/

DMA_t DMA;

/*
* Private data
*/
 const Pin pinDmaAcknowledge = PIN_DMA_ACK;
 const Pin pinDmaRequest = PIN_DMA_REQ;

/**********************************************************************\
*       Public functions
\**********************************************************************/
void DMA_Initialise(void)
{
    DMA.Mode   = 0;
    DMA.Memory = 0;
    PIO_Configure(&pinDmaRequest, 1);
    PIO_Configure(&pinDmaAcknowledge, 1);
    PIO_ConfigureIt(&pinDmaRequest, (void (*)(const Pin *))DMA_Request_ISR);
}

void DMA_EnableInterrupt(void)
{
    PIO_EnableIt(&pinDmaRequest);
}

void DMA_Set_Mode(uint8_t mode)
{
    DMA.Mode = mode;
}

uint8_t DMA_Get_Mode(void)
{
    return DMA.Mode;
}

void DMA_Set_Memory(uint16_t *value)
{
    DMA.Memory = value;
}

void DMA_Set_Peripheral0(uint16_t volatile *value)
{
    DMA.Peripheral = value;
    DMA.Peripheral8 = (uint8_t *)value;
}


/**********************************************************************\
*       Private functions
\**********************************************************************/

#define PIO_PDSR        (AT91_CAST(AT91_REG *) 0x0000003C) // (PIO_PDSR) Pin Data Status Register
#define PIO_SODR        (AT91_CAST(AT91_REG *) 0x00000030) // (PIO_SODR) Set Output Data Register
#define PIO_CODR        (AT91_CAST(AT91_REG *) 0x00000034) // (PIO_CODR) Clear Output Data Register

#define PORTC_READ     *((volatile unsigned long*)(((int)AT91C_BASE_PIOC + (int)PIO_PDSR)))
#define PIN_DMA_REQEST_MSK     (1 << 13)

#define PORTC_SET     *((volatile unsigned long*)(((int)AT91C_BASE_PIOC + (int)PIO_SODR)))
#define PORTC_CLEAR     *((volatile unsigned long*)(((int)AT91C_BASE_PIOC + (int)PIO_CODR)))
#define PIN_DMA_ACK_MSK (1 << 1)

/**
 * \brief   DMA Interrupt handler
 * \details optimised to use port registers directly
 *          Peripheral addresses don't change so pointer stays fixed
 *          Memory addresses do chage so pointer increments
 */
static void DMA_Request_ISR(void)
{
    if(DMA.Mode & DMA_W_WORD)
        DMA_Process_16Bit();
    else
        DMA_Process_8Bit(); // this is almost identical to 16Bit
}

static void DMA_Process_16Bit(void)
{
    uint16_t *ptr = DMA.Memory;
    uint16_t volatile * periph   = (uint16_t *)DMA.Peripheral;
    uint32_t volatile * Request  = (uint32_t *)&PORTC_READ;
    uint32_t volatile * SetAck   = (uint32_t *)&PORTC_SET;
    uint32_t volatile * ClearAck = (uint32_t *)&PORTC_CLEAR;

    if((DMA.Mode & 0x60) == DMA_MODE_IO2MEM)
    {
        while(*Request & PIN_DMA_REQEST_MSK)
        {
            *ClearAck = PIN_DMA_ACK_MSK;
            *ptr++ = *periph;
            *SetAck = PIN_DMA_ACK_MSK;
        }
    }
    else // must be mem to IO then
    {
        while(*Request & PIN_DMA_REQEST_MSK)
        {
            *ClearAck = PIN_DMA_ACK_MSK;
            *periph = *ptr++;
            *SetAck = PIN_DMA_ACK_MSK;
        }
    }
    DMA.Memory = ptr;
}

/**********************************************************************\
   End of file Dma.c
\**********************************************************************/

It looks like I did have to optimize it afterall. It uses the Atmel Library to set up and then the registers directly

[Jeroen3]

NXP has dual core parts where you can use the extra M0 as dma. Some are flashless, so I'd expect the external bus behavior to be well documented.

[westfw]

I doesn't surprise me that any modern external memory controller optimizes for consecutive reads/writes at the expense of "turnaround" time.  After all, cache-fill, fast page mode, etc, etc.  It does surprise me that this isn't documented carefully (although I guess it falls between the pieces that ARM would document (bus) and the parts that the individual vendor would document (memory controller.  Maybe?  Is that one of the pieces you can buy from ARM as well?)

Did you see this bit in the datasheet:

[size=0pt]22.9.4 Read to Write Wait State [/size]
[size=0pt]Due to an internal mechanism, a wait cycle is always inserted between consecutive read and write SMC accesses. [/size]
[size=0pt]This wait cycle is referred to as a read to write wait state in this document. [/size]
[size=0pt]This wait cycle is applied in addition to chip select and reload user configuration wait states when they are to be inserted. See [/size][size=0pt][color=rgb(0.000000%, 56.500000%, 100.000000%)]Figure 22-16 on page 207[/color][/size][size=0pt]. [/size]
     

[SimonR]

Thanks westfw

Yes I did see that bit of the datasheet, unfortunately not before the board was designed. It was not the highest priority to fix and luckily it still just about meets the spec.

The actual question in this thread is to ask if there is another chip I can use that doesn't have this problem. The next version needs to be faster but I am still stuck with the demand driven DMA architecture.

Further down the line there will be an FPGA solution but not this time around as the requirements are not yet set.

[Jeroen3]

Is that single wait state that critical? If so, isn't it better to get an application processor that is running faster than 180 MHz?
Such as the category used on a raspberry pi.

[SimonR]

Jeroen I fear you may be right.

That single wait state is unbelievably critical.

The irony here is that the ARM replaced a 1980s 16 bit microcontroller running at 20MHz and it only had a fixed 100nS bus cycle itself.
It seems total madness that I might have to go to 1GHz processor just to get a fast enough bus interface.
I don't want to as I need on board flash as I don't have the board space.

[Jeroen3]

You know you can use quad spi flash? That is relatively fast. And if you copy the right parts of code to ram, even faster.

Meanwhile, can you overclock the EBI?

[nctnico]

An alternative is using SD cards since many microcontrollers have a dedicated SD card controller which (most) probably can use DMA.

[SimonR]

Its not flash speed thats the problem its board space.  quad flatpaks take quite a bit of space. Going to BGA is not a desirable option in this case because it changes the production process and adds 2 more layers to the PCB.

Overclocking the EBI? I hadn't thought of that, I'll investigate.

SD cards might be an option later, its already on the list but we need to be backward compatible this time so we are stuck with CF cards.

[knielsen]

A number of the STM32 MCUs (STM32F469 for example) have a core-coupled SRAM memory. This is not on the AHB or bus matrix, but connected directly to the Cortex M4 core.

So if the DMA code is put in the core-coupled SRAM, along with the intermediate buffer, I'm guessing that arbitrary amounts of words could be buffered between switching direction (well, up to 64 kByte).

However, I'm a bit confused - you write that you can do back-to-back reads at full speed (and similar writes) and buffer in registers. But your code also does a read/write switch per word to handle REQ/ACK - doesn't that prevent the speedup from buffering? Or is it only when the external ram controller switches direction that you get the penalty?

I any case, it must be the external memory controller that has the penalty from switching direction, right? At least I am pretty sure that the high-end STM32's (that I am most familiar with) can easily read/write access internal busses much faster than that.

If you are interested, I have a custom board with an STM32F405 and an external SRAM on the FSMC (ST's external memory controller). I could do some testing to check the speed of reading/writing when switching direction?

Also, the FSMC has some extra signals available, like an external WAIT signal and such. Maybe your MCU has something similar, perhaps with a bit of trickery those can be used to handle the REQ/ACK protocol internally in the controller?

[SimonR]

Hi knielsen

Thanks for the offer, theres nothing like having proof that something works before buying it.

Sorry for the confusion.
Yes my actual production code does READ WRITE READ WRITE because of the demand driven DMA problem.

I wrote some seperate test code that did 4 reads in a row just to prove the point. This was from the CF card as I can't set up a test from the DMA peripheral.

I did consider writing to internal SRAM and then to external CF (and the otherway round ) but my collegue reminded me that the wait state still applied to internal RAM as well.
Its about a year since the test so I need to run this test agian to confirm it. It may be true because I remember swearing heavily at the time.

I'll see if I can find my logic analyser traces, as I think they may be helpful.

[SimonR]

I found the email trail to ATMEL which I'll sumarize later but here are the actual timings.
Basically Atmel confirmed that its a bus matrix problem, when the master in controll of it changes.

The exact code running here is:-

Code: [Select]

*periph;                //0x204024  0xe1d230b0  LDRH    R3,[R2]   ; this always reads because its volatile before any C programmers complain
*buffer++ = *periph;    //0x204028  0xe1d230b0  LDRH    R3,[R2]
                        //0x20402C  0xe0c030b2  STRH    R3,[R0], #0x02 

And its running out of cache so has zero impact on bus timing.

The timing for the *periph; section is the first trace
and *buffer++ = *periph; is the second.

[Jeroen3]

A number of the STM32 MCUs (STM32F469 for example) have a core-coupled SRAM memory. This is not on the AHB or bus matrix, but connected directly to the Cortex M4 core.

Only the D-bus of the cortex m4 has access to this. No instructions and no dma access to there.

[SimonR]

the ST32 may still solve the problem if its not connected to any bus matrix type architecture.

Also if anyones wondering why this is such a problem, its because we sometimes access the whole card in a single transaction.
That means that for 64Gig card it adds an extra 82 minutes. 144 nS * 32Gig Reads  = 4947 seconds.
Its only 5 minutes on a 4Gig card which is why its not a problem now. But when the card sizes increase it will become a problem.
(it is a problem but we can live with it)

[knielsen]

Ok, so this was quite interesting. I made a quick ('n dirty) test program
using an STM32F405 MCU connected to an external 16-bit SRAM IC. The MCU is
running at 168 MHz (6 ns CPU cycle). The SRAM has 55 ns access time.

The FSMC (memory controller) is configured like this:

 1  BUSTURN  delay after read
 6  DATAST   data-phase duration
 0  ADDHLD   address-hold phase duration
 3  ADDSET   address setup phase duration

That is something like 10 CPU cycles per FSMC transaction (60 ns to match
the 55 ns access time), plus any overhead. The exact timing would need to be
determined for the requirements of your specific peripherals, of course.

I tested single and burst read, single and burst write, read followed by
write, and GPIO read + SRAM read + SRAM write + GPIO write (this last one to
resemble your DMA code). For example:

  do
  {
    (void)*gpio_read_ptr;
    (void)*sram_ptr;
    *sram_ptr = some_value;
    *gpio_set_ptr = gpio_set_val;
    -- i;
  } while (i != 0);

The results show that an external SRAM read or write takes 12 CPU cycles, or
72 ns, in all the cases basically. I see no penalty for changing direction
on the bus. In fact, the overhead of adding the GPIO read/write is exactly
zero cycles! I believe this is because the FSMC has a CPU cycle or two of
delay between transactions, which can be overlapped with the GPIO accesses.

Put another way, I am getting 14 MByte/second bandwidth copying external ram
to external ram on the STM32F405, with this particular SRAM IC.

So basically, the ST's Cortex M4 in my tests seem to be able to communicate
on the external bus at the full configured speed. Just note that the M4
usually takes 2 CPU cycles (12 ns) to do a store or a load (plus any wait
states), which sometimes can go to a single cycle if load and stores are
scheduled optimally in the code.

But I noticed that the at91Sam9XE you are using seems to be a full ARM
application processor with MMU and such? The STM32F405 is a Cortex M4
microcontroller, it does not have MMU or caches or other such things...
These ARM chips are designed specifically with a mind to get fast and
predictable timing for things like peripheral access and interupts and such,
which might explain the difference...

If you are interested, you can find my test code on my github (note the
non-default branch 'eevblog', I hacked an old test program I had):

  https://github.com/knielsen/stm32f4_sram_memtest/tree/eevblog

Hope this helps,

 - Kristian.