32F417 USB FS (not HS) and DMA, and how to write via USB to a slow FLASH drive

[peter-h]

Is it possible to replace this loop with DMA?

Not even sure the 32F417 DMA can connect to that USB FIFO. No mention in the RM. There is a dedicated DMA for the USB (like there is for ETH, where you really need it) but is it not available in FS mode (only in HS mode).

But if the USB FIFO has an address then surely one could do a memory-memory transfer? There is no "busy" status.

[ataradov]

Regular DMA should be able to write this FIFO. But if you are not using isochronous transfers,the maximum packet size is 64 bytes (16 words). It is not worth it to setup DMA for that short of a time.

[peter-h]

Is FS mode never more than 16 words? That indeed would not be worth doing anything with. Although one could have a DMA channel set up and just load the transfer count and enable it to do the transfer.

[ataradov]

Only isochronous transfers can go up to 1023 bytes (~256 words). All other transfers are 64 bytes maximum.

But then you also need to poll for completion status. With time spent preparing and starting DMA transfer and waiting for it to complete, it will be a wash.

[peter-h]

I am wondering what actually happens if a Windows machine is accessing an embedded target like mine via USB.

I am running a VCP for debugs; not relevant.
I am also running the block device, which does 512 byte sector r/w from a serial FLASH. I've been optimising this part, as per the other thread. The writing will always be slow but that's OK, and I got the reading down to 200us for a sector. But if the USB comms uses just 64 byte packets, in the case of a write:

Windows must be sending 8 of these in succession, which get placed into some buffer (the USB code has some which are a few k), all under interrupts (which I assume must be separately serviced) and then issues a "write", and that generates the flash write.

In the case of a Read, Windows must request a sector read, and then issue 8 polls to retrieve the 512 bytes.

Is the above about right?

FWIW, I am reading a 2MB file from FatFS based 21mbps SPI serial FLASH in about 3 seconds (this is doing
copy /b i:*.* nul
in a DOS box in win7-64, obviously taking care to not be reading cached data) which is pretty respectable, but I am more concerned for how long the target interrupts are blocked, which they seem to be comprehensively by USB interrupts

[ataradov]

Windows must be sending 8 of these in succession, which get placed into some buffer

Each of the 8 messages is received separately on a hardware level. Your USB stack reads and accumulates all of them. It depends on your USB stack API, but usually you provide a buffer to store the data. This buffer must be big enough to store 512 bytes. USB stack will reassemble the parts and provide a completion callback for the whole buffer.

In the case of a Read, Windows must request a sector read, and then issue 8 polls to retrieve the 512 bytes.

Same thing. You provide a buffer and submit the request. The contents of the buffer are sent in 8 packets, and the common callback is called for the whole buffer.

target interrupts are blocked, which they seem to be comprehensively by USB interrupts

It is hard to tell what specific stack implementation does, but there is generally no need to lock the interrupts for too long. That fill of the FIFO is pretty fast. The only possible issue here is that if completion callback is called from the interrupt handler, then you should not do any real work in that callback. Mark the buffer as ready to write and return. Then in the main loop check for the flag and write the data. Again, you need to know how USB stack is implemented to write optimal code.

[peter-h]

"Mark the buffer as ready to write and return"

Sure; that was the advice in the other place
https://www.eevblog.com/forum/microcontrollers/cleanest-way-to-block-usb-interrupts/msg3949951/#msg3949951

but it still leaves the question of how to stop the host pushing in more writes. There has to be a way of returning a BUSY to the host, until that 512 byte sector has been written to the FLASH and the FLASH program cycle has finished.

I am using the STM32 USB device library from ST, as supplied with Cube. Someone spent some months getting that to work But it seems to work well.

[ataradov]

but it still leaves the question of how to stop the host pushing in more writes. There has to be a way of returning a BUSY to the host, until that 512 byte sector has been written to the FLASH and the FLASH program cycle has finished.

It happens automatically. When the host tries to send you new data and there is no buffer to receive that data (endpoint is not armed), USB controller would return NAK and the host will try again. And the only time the data would be accepted is when your applications request a read again.

[peter-h]

How is the endpoint armed (or not)?

[Tagli]

AFAIK, OTG_FS can hold up to 7 packages in its endpoint TX FIFO. So, it could make sense to load 64 x 7 (448) bytes for bulk transfer using DMA. But there are some issues:

1) I'm not sure if DMA can access FIFO push register.
2) I think classical 64 x 2 double buffer is easier to manage in software.
3) OTG_FS can't make DMA requests. So how to trigger DMA? Maybe one can use interrupts to trigger it, but just performing 8 x 32-bit writes is easier. No need to use DMA for that.

For RX, it would be harder I guess. Because there is only one RX buffer, and hardware itself pushes some metadata into RX buffer, along with the actual USB data.

BTW, I have no idea how OTG_HS handles its dedicated DMA. I haven't read that part of the RM.

[ataradov]

How is the endpoint armed (or not)?

No idea in case of STM32. See what your firmware does when reading stuff. Based on the name of the function you posted, see what USB_ReadPacket() does. Or read the reference manual. 

[peter-h]

This is both of them

Code: [Select]

/**
  * @brief  USB_WritePacket : Writes a packet into the Tx FIFO associated
  *         with the EP/channel
  * @param  USBx  Selected device
  * @param  src   pointer to source buffer
  * @param  ch_ep_num  endpoint or host channel number
  * @param  len  Number of bytes to write
  * @param  dma USB dma enabled or disabled
  *          This parameter can be one of these values:
  *           0 : DMA feature not used
  *           1 : DMA feature used
  * @retval HAL status
  */
HAL_StatusTypeDef USB_WritePacket(USB_OTG_GlobalTypeDef *USBx, uint8_t *src, uint8_t ch_ep_num, uint16_t len, uint8_t dma)
{
  uint32_t USBx_BASE = (uint32_t)USBx;
  uint32_t *pSrc = (uint32_t *)src;
  uint32_t count32b, i;

  if (dma == 0U)
  {
    count32b = ((uint32_t)len + 3U) / 4U;
    for (i = 0U; i < count32b; i++)
    {
      USBx_DFIFO((uint32_t)ch_ep_num) = *((__packed uint32_t *)pSrc);
      pSrc++;
    }
  }

  return HAL_OK;
}

/**
  * @brief  USB_ReadPacket : read a packet from the Tx FIFO associated
  *         with the EP/channel
  * @param  USBx  Selected device
  * @param  dest  source pointer
  * @param  len  Number of bytes to read
  * @param  dma USB dma enabled or disabled
  *          This parameter can be one of these values:
  *           0 : DMA feature not used
  *           1 : DMA feature used
  * @retval pointer to destination buffer
  */
void *USB_ReadPacket(USB_OTG_GlobalTypeDef *USBx, uint8_t *dest, uint16_t len)
{
  uint32_t USBx_BASE = (uint32_t)USBx;
  uint32_t *pDest = (uint32_t *)dest;
  uint32_t i;
  uint32_t count32b = ((uint32_t)len + 3U) / 4U;

  for (i = 0U; i < count32b; i++)
  {
    *(__packed uint32_t *)pDest = USBx_DFIFO(0U);
    pDest++;
  }

  return ((void *)pDest);
}


[ataradov]

Ok, I read the manual for you.

In case of STM32 USB controller would automatically ACK all incoming frames as long as there is space in FIFO. You can also enable NAK on all out transfers using "Global OUT NAK", but this is not useful here.

So, the endpoint is always armed as long as there is space in FIFO. As soon as space runs out, controller would be sending NAKs until you read the data.

[Tagli]

It's also possible to enable NAK on a specific endpoint. So, you can stop reception even when you have free space on RX FIFO.

[ataradov]

Yes, I did not find that on a quick read, and it made no sense to me, but now I see there is a control of this via SNAK / CNAK bits in the OUT endpoint control register.

[peter-h]

So you think that one can simply stop usb_read_packet while the flash is programming, and that will take care of it?

[ataradov]

Yes, sure, but it is a very bad solution. It is generally a good idea to receive full sector (512 bytes) into the temporary buffer as fast as possible, and then write it to the flash from the main task.  The actual write status is communicated over separate MSC commands, your application should not normally even call USB_ReadPacket(), it would be done by the MSC stack, which will give you a full payload. You should not block USB stack from working while write is happening.

[peter-h]

Sure; understood. I am, as you can see, without a full understanding of this incredibly complex subsystem, and I am trying to work out where to do what.

USB_ReadPacket() is called by ST code, not by my code. It's a part of the stack, AFAICT.

Eventually the stack calls this

Code: [Select]

/**
  * @brief  .
  * @param  lun: .
  * @retval USBD_OK if all operations are OK else USBD_FAIL
  * The page write is about 18ms.
  *
  */
int8_t STORAGE_Write_FS(uint8_t lun, uint8_t *buf, uint32_t blk_addr, uint16_t blk_len)
{
#ifndef USB_INT_PROTECT
if (flash_busy) return (USBD_BUSY);
#endif
Flash_WritePage(buf, blk_len * STORAGE_BLK_SIZ, blk_addr * STORAGE_BLK_SIZ);
return (USBD_OK);

}

but the line return (USBD_BUSY); didn't work. It sounds like this function should just set a "got a buffer, write it to flash" flag (which gets an RTOS task to flash the buffer) and exits with an OK status immediately. But if the flash is busy programming (e.g. still programming the last sector), what is it to do? It can't exit with "USBD_OK" immediately because the host will just send another sector.

Incidentally, as reported previously, the calling code didn't support any return status other than USBD_OK But even after that bug was fixed, it still didn't work. So I guess ST's code was never tested with a storage device other than something fast enough to be fully written inside that ISR.

[ataradov]

I don't know how their APIs are supposed to work. My personal preference is to spend a bit of time upfront, but end up with something I understand and know how it works exactly.

[peter-h]

It looks like there is really no point in DMA for the 16 words, but it would be great if somebody knew how to indicate to the host that the storage device is busy.

[ataradov]

In USB MSC protocol host does not simply push the data. The transfer starts with a Command Block Wrapper (CBW), then data is transferred, which is followed by the Command Status Wrapper (CSW).

The host will not send any new commands or data until the status of the last command is received. Quote from the spec:

The host shall not transfer a CBW to the device until the host has received the CSW for any outstanding CBW. If the host issues two consecutive CBWs without an intervening CSW or reset, the device response to the second CBW is indeterminate.

So all you need to do is figure out how to delay transmission of the CSW until you are ready to get more data, and send it when you are ready.

Check that there is not some status code that STORAGE_Write_FS() can return, which would cause the CSW to not be sent. And then there must be some API that would separately send CSW.

And if not - then here is your simplest solution. Modify STORAGE_Write_FS() to not send CSW and add that API to send it when you are done.

[peter-h]

OK thank you. Will look into this.

The funny thing was that STORAGE_Write_FS returns only "OK" (which here is 0) and the other codes available are

Code: [Select]

/* Following USB Device status */
typedef enum {
  USBD_OK   = 0U,
  USBD_BUSY,
  USBD_EMEM,
  USBD_FAIL,
}USBD_StatusTypeDef;

which are 0 1 2 3

while the code calling STORAGE_Write_FS would accept only 0 and then negative numbers So anything other than "OK" was never tested. Unfortunately modding the calling code (which gets to STORAGE_Write_FS via this function table):

Code: [Select]

USBD_StorageTypeDef USBD_Storage_Interface_fops_FS =
{
  STORAGE_Init_FS,
  STORAGE_GetCapacity_FS,
  STORAGE_IsReady_FS,
  STORAGE_IsWriteProtected_FS,
  STORAGE_Read_FS,
  STORAGE_Write_FS,
  STORAGE_GetMaxLun_FS,
  (int8_t *)STORAGE_Inquirydata_FS
};

to accept a BUSY, and getting STORAGE_Write_FS to return a BUSY, was not successful.

I will have another dig around.

[peter-h]

Looking at this some more, it looks like there is an interrupt when the FIFO is full.

This then triggers the software copy loop, which free runs, without any status checking, moving (presumably) 64 bytes, into the 512 byte buffer.

Then the USB stack requests more data and gets another 64, so another FIFO-full interrupt, and another 64 bytes copied, further along into the 64 byte buffer.

Then, when all 512 bytes have arrived into the 512 byte buffer, the USB stack calls that final function which does the FLASH write.

All the above is an interrupt driven state machine.

IF the assumption is correct that the USB stack is using the FIFO status to request more data, an easy approach might be to disable this interrupt and instead poll that status bit (if there is one) from an RTOS task, and implement the FLASH programming from within that. That way the FIFO will be left full during programming and it all "should just work".

The problem is that interrupt runs a long chain of things...

[ataradov]

This is not correct. As I said, after you do your flash write, you should see MSC driver send CSW. Your goal is to prevent transmission of that CSW. If you do this, there will never be any data in the FIFO until you sent it. This is the only correct way to approach this. Anything else is a hack that may or may not work and may or may not be compatible with different OSes.

[peter-h]

"after you do your flash write"

Do you mean "before you commence your flash write"? And then send the CSW when the flash write has finished?

That's probably quite a rewrite of the ST USB stack. I bet it's been done by many, and unpublished because this has to be a common requirement. You can read flash very fast, down to ~50us for a sector, but no way to write it fast.

OTOH, the USBD_OK return may be what sends back the CSW, so not doing anything in that lowest level function (where currently the flash write is done) might do it. Then I need to find a way to send the CSW when the flash write ends.