Reading Data Over Serial & Storing in Char Array for Processing?

[Dumble]

Hi,

I am using an Atmega328 Standalone and a GSM module in my project but I've hit a bit of a brick wall with my software program when it comes to reading in an SMS message and processing it in the way I would like.

It doesn't help that I'm so bloody rusty at C that I didn't even realise how rusty I was until I set about this project.... I've not done any real C since back in College around 10 years ago. Still, you know what they say? "Use it or lose it".


What I am Trying to Do:-

Anyway, I am trying to develop a function for reading the last SMS message received  which should be stored in the SIM's memory and then process the commands contained in the SMS Message Body BUT, I'm now getting to the point where I'm going round in circles. I've been over loads of forum posts, websites and example codes this past week and all the knowledge is there but I just can't seem to get it down in the form of an efficient function that does what I want.

My project has around a dozen commands which a user can send via SMS which the Atmega should process and act on. I want to be able to send those commands via SMS as either a single command or multiple commands of mixed data types in no particular order.

E.G. A Single Command: MCT0 would be Master Control Off. OR, MCT1 would be Master Control On. The variable storing this setting is a Global Bool Variable which should get updated with the new setting.

E.G. Multiple Commands: MCT0, txF22.0000, sQL5  OR in a different order: sQL5, txF22.0000, MCT0


What I've tried:-

I've been all over the internet this past week looking at example codes, reading forum posts, websites and Q&A websites and find bits and pieces of information but I just can't settle on the best and most efficient way of doing this.

Just when I think I've found an idea there's usually there turns out to be some glaring flaw with it such as, it' turns out to be inefficient, it's out of date, it doesn't work with my GSM module or several other reasons which stop me in my tracks and force me to re-think my approach.

For all the reading, I feel like I've gained all the information I need to do it, I'm just struggling with how to put it altogether to make a logical and efficient function. To be honest, I feel like I've got writers block, if that makes sense? All the words are there but it all just comes out like cr*p when I try to get it down lol


So I would really appreciate some guidance with this one if at all possible. I don't mean I want the code writing for me but I would like some help, advice, hints and tips to get me in the right direction and to help keep me on track while I try to nail this thing...


Hopefully, with help, once I get back into the swing of things, it'll start coming back to me and I'll find my flow again.


Sorry for the long post but I try to be descriptive and thorough.


Thanks!

[alexyzhov]

I choose to use array buffer to do asynchronous processing, otherwise interrupt would take sooo much CPU time

[Buriedcode]

Well splitting it into two main tasks would help:

1) read the message, and check if there are any commands, this can be done in an interrupt - just sets a flag to say if one or more commands are there.
2) if above flag is set... loop through message to find the commands, raising a flag for a particular command when it is there.  So each command has its own flag here.
3) BY checking those flags one by one, for each command, do what that command does (that sentence reads like a Forest Gump Philosophy)

In any case, you'll need to read in the text and store in a buffer... unless... you simply read it in, one character at a time, and as you go, flag up any commands, along with storing their parameters.
That might require less RAM, with the penalty being you spend more time in your interrupt, which may or may not increase servicing latency for other tasks.

[rstofer]

First of all, I would receive the chars and put them in a circular buffer by way of an interrupt handler.
As you receive chars, you could set a flag when you receive the end of a command.

Your queue needs to be large enough to handle the incoming command rates OR you need to implement flow control and that might be difficult.  X-ON, X-OFF is the software approach.  RTS,CTS (hardware) is another.

The interrupt routine could recognize the end of a command sequence and put it into another queue for the mainline.  This internal form may be encoded in a more suitable fashion.  A byte (number) for the command, a byte for the operand, etc.

If your mainline is fast enough, you can grab the chars from the UART (or a smaller interrupt driven queue) and assemble the commands on the fly.  This is probably the easiest.  Your UART interrupts, the interrupt handler grabs a char and stuffs it in a queue.  Your mainline grabs chars from the queue on every iteration and builds the command string.  Eventually the code sees the end of a command sequence and executes it.  Your character queue doesn't need to be any bigger than the maximum number of chars that could occur during the longest pass through the mainline (including completing the command and executing it).

http://pcarduino.blogspot.com/2013/10/atmega328s-usart.html

[westfw]

The usual technique is to have an interrupt-driven character oriented buffer that reads the uart, and code at user level that reads a a character at a time into a "message" buffer (eg "getline"), and then parse that buffer as needed.  This is portable to desktops and other languages and full operating systems (where "getline" may be built into the operating system), and is usually efficient enough in CPU and space requirements for most applications.
It gets harder if messages are bigger than your RAM, or if your data is not at all easily separated into messages, or needs extraordinarily quick response, in which case you can start thinking about parsing the message at interrupt level and only storing the "important bits", but that's usually overkill.

[Dumble]

Hi again and thank you to all for replying with help, advice and suggestions!
Any sort of help really is appreciated!

Yesterday, writing out my Original Post seemed to give me a little more direction and clarity so I decided to go back and re-visit a technique I picked up on my Systems Analysis and Design course - Pseudo Code. Why didn't I think of laying it out like that before? DOH!

Actually, Systems Analysis and Design was another of my favorite courses back in College. There was something satisfying about creating all the diagrams, identifying the data and data flow etc that I really enjoyed. I can remember the first day on the course and our lecturer trying to explain One to Many and many to many relationships using a public library database as the example. Boy that was fun thrashing that out hehe.

Anyway, I digress with reminiscent ramble.... LOL So I was saying before I wandered off on a tangent down memory lane like some senile and eccentric gentry...

I've roughed out some Pseudo code to help identify what needs to happen, when it needs to happen and roughly how it needs to happen etc. which I have attached to this post in a notepad txt doc for anyone that might want to take a look.


So on to a couple of questions and responses if I may?..

Obviously the first thing I need to do is to get the data from the GSM module into the Atmega for processing.

Circular Buffer:-

I've not heard the term circular buffer before or maybe I have but I know it as something else or have used it without knowing it to be a circular buffer?

I've just done a quick read on the internet and I think I understand. It's just a buffer that can be overwritten and I think the most common form of that I've seen is used in examples I've looked at for reading in serial data is using a single char to read serial data character by character and then storing that in a location inside of an array of char using a position keeping index variable.


Something roughly along the lines of the code below which is based on / taken from an example located here: http://forum.arduino.cc/index.php?topic=89909.msg674783#msg674783

Code: [Select]

char msgData[90]; //Message Buffer Holds Entire Message Content char by char
int index = 0;

setup(){
Serial.begin(9600);
}

loop(){

while(Serial.available() > 0){
char readChar = Serial.read();  // Circular Buffer?

if(index <89){
msgData[index] = readChar;
// These lines will keep adding /0  to the array AFTER each character has been stored until the array is full leaving /0 as the last item in the array
index++;                             // Advance to the next index position in the msgData array -
msgData[index] = '/0';        // Write /0 NULL terminator to the specified index location
}

}

}

Assuming that's roughly right, that should leave me with an array of char msgData[90] containing all of the SMS content including the time and date stamp, senders number etc. ready for sorting and processing in the next steps?

Now in the actual example code on the website in the link above, it uses Start of Packet and End of Packet indicators / markers (specific characters such as "<" and ">") defined in the code to signify the beginning and end of the SMS data that's relevant. But before I get to the SMS message content or packet, I need to locate the sender number which is sent first in the header along with some other information such as time and date stamp.

So the first thing I need to extract and verify is the sender number and make sure it is valid to issue commands to the system. Pending of course that the SMS message that's been read in and stored contains valid data and is free from errors or garbage.....

Which brings me to....


Error Checking:-

One thing that I have made no provisions for yet is error checking the SMS messages and I think now might be a good idea to address that as the number one process after initially reading in and storing the SMS message.

It is quite possible that while reading in serial data, something could screw it up and lead to garbage coming across and being stored.

The only way I can think to get round this at the minute is to read the same SMS message in twice, store each message in to its own array and then compare the data in the two arrays to ensure they are exactly the same. I would expect the data in both arrays to be identical if there have been no problems with reading the serial data. If the content of the two arrays don't match, read the SMS again again until they do.

I'm sure its quite possible to go overboard with error checking but I just need something basic to verify the data is good and this seems the most basic way.


I think I'll stop at this point with my questions before I get waaay ahead of myself and confuse myself again and anyone else reading this.


My goal for today is obviously to get the data acquisition and storage part of my Read SMS function written and working. Then I can start worrying about what to do with the data once I have it stored.


Thanks again!!!

[Buriedcode]

It is quite possible that while reading in serial data, something could screw it up and lead to garbage coming across and being stored.

How so?  Whilst error detection/correction is pretty much required in low level wireless applications, your GSM will do the over-the-air error handling.  The only possible errors you could get are between the GSM module and microcontroller.  Unless there is a long cable, a bad connection, massive interference, or mismatched clocks, errors on a serial line are rare.  I wouldn't worry about checking the serial data for errors - unless you mean typo's in your SMS commands?  In which case reading it in twice won't make a difference.

I understand your thinking - "what if" scenarios are good to try and catch possible problems with robustness before they start, but error checking the serial line is most likely a waste of time.

First things first - get something working.   A few of has have provided a rough outline:
- Interrupts to handle reading in the data, setting a flag for the main routine when you reach the end of the message.
- Main routine scans through message to find commands, and either processes them as it finds them, or sets flags (one for each command) and stores data for them
- then, if you haven't done so, go through each command.

So to start with, the interrupt routine.  You could create a second software UART on your AVR as debug, that spits back out the SMS it just read in, to check that the interrupt works, and it raises a flag once it has got to the end of the message.
Then a routine to check for "commands".  Are you deciding on the format of these commands or are they some sort of standard?  Assuming that each command starts on a new line, you can use the CR (carriage return 0x0D) character as a place to check for known commands.
Then of course is just a routine for each command.

I think many of us here have done something similar - perhaps with varying complexity - and there is more than one way to go about it, as you've found.  With an 8-bit micro, things like message queues, whilst robust, might be rather bloated.  But then again the '328 has a whopping 32KB of flash.

Re: circular buffer.  A circular buffer has two pointers, or 'heads' as some people call them.  One write pointer, which points to the location where data is to be written, and one read pointer.  When you write to it, you simply write a value to the array entry the write pointer is pointing to, then increment the write pointer.  And with reading, you read the value that the read pointer is pointing to, then increment the pointer.  There are ways to handle over-run and under-run, as well as bounds (if your array has 16 entries, values 0 to 15, then you want to make sure your value is within that).  It can get quite complicated, but often they are fairly simple.

In your case, if you can afford the RAM usage - assuming your 90 character limit - I would just read in the entire message, perhaps store the message size.  If you're expecting to receive a message whilst handling the previous one, then a ping pong buffer could be used.  That simply means, one buffer is used to read in a message, once it has been read, the micro flags that buffer to be the one to be read, whilst falging the other buffer to be used to read in another message.  So it ping pongs between buffers - you can read in another message, whilst handling the other.  With 2KB of SRAM on the atmega, 180 bytes isn't much for the buffer.

[NorthGuy]

Anyone can send an SMS to your device. And now, when you published the interface, anyone can control it.

[AndyC_772]

The first question I'd be asking myself is whether there's a truly compelling reason to be using an 8-bit CPU at all, when there are so many more capable options out there. I started out learning microcontrollers using a PIC18, but now I use STM32 processors unless there's a compelling hardware reason not to (eg. unique peripherals, voltage range, temperature tolerance and the like).

To handle incoming serial commands, I first set up a receive buffer, and program a DMA controller in circular mode to capture all incoming serial traffic into it. This ensures that all received characters are stored without wasting time processing an interrupt for each and every one.

Then, at regular intervals, I check the DMA controller to see how many characters have been received since the last time round the loop. If there are any, I copy them into a separate buffer which holds a complete command line. This means the parser doesn't have to cope with wrap-around at the end of the DMA buffer. I could also take the opportunity to filter out invalid characters, and to ignore any line which contains them.

Finally, when a command is deemed to be completely received (ie. a CR/LF), the contents of the command buffer are parsed. Any remaining characters in the DMA buffer can simply be left there, as they'll be retrieved next time round the loop.

This method requires enough RAM to hold a complete maximum-length command line, plus as many characters as may ever be received in between times round the main loop - probably not that many in most cases, and you'd need somewhere to put received characters anyway when you're busy parsing a previous command. Major plus points are that there is no need for any code whatsoever to run each time a character comes in, and unless the receive buffer is too small, no incoming characters can ever get lost, however busy the CPU might be.

Downside? You can't do it on a CPU that lacks a DMA controller.

[rstofer]

The linear array for storage of incoming characters works well as you assemble command strings.  One assumes that the first char of the array is the first char of the incoming string.  But you need a strategy for flipping between buffers processing one that is complete while allowing the interrupt handler to insert incoming chars in the other.  This approach has the interrupt handler involved with determining the start and end of command strings.

The circular buffer doesn't care about the lexical information, it simply puts chars in a queue.  Some other piece of code takes chars out of the queue and parses them for command strings.  The important thing about interrupt routines is to get in and get out fast while doing no more than what is absolutely required.  Interrupt routines interrupt everything else that is going on and if there is some bit of code somewhere else that is time sensitive, the last thing you want is to be messing around in an interrupt handler.

Embedded processors usually have a super loop:

while (1)
{
...
}

At the top of the loop, you call a function that grabs chars (as many as it can, one at a time) and builds strings.  When it has a complete command string it might set a flag to tell the super loop to handle the command.  Maybe it looks like this:

while (1)
{
  if (0 != getcommand() ) // returns 1 if command available, 0 otherwise
  {
    // we have a command
    processcommand(); // somehow this routine needs to know where getcommand() put the command string
  }
  .. other stuff in super loop goes here
}

getcommand just uses a dequeue function (the interrupt handler uses the enqueue function) to grab the available chars and it is called on every cycle through the loop.  Eventually, getcommand recognizes the end of a command string and returns non-zero.  There is now a command string in a buffer.  There will probably need to be a data structure (array of <something>) where getcommand can put the assembled command.  I usually build up some kind of struct and then organize a circular queue of structs.  In this case, the super loop just trys to dequeue a command struct.  If successful then it can execute an already decoded command from the struct.  If there is nothing in the queue, there is no command to execute.

So, maybe two queues:  One to hold incoming chars and one to hold assembled/decoded commands.

It's important to lay out the data flow from incoming chars through executed commands and see which operations belong to which functions.  Who grabs the chars from the UART and enqueues them.  Who dequeues the chars and builds commands.  Does this function maintain a queue of command structs?  Who dequeues the command structs and executes the command?  How long does this process take?  Is there time left over  when thinking about the incoming data rate and queue sizes?

Another way of thinking about getcommand() is that it gets chars from a queue, assembles command structs and, when the command string is complete, puts the assembled struct into a command queue for processing by some other bit of code which dequeues the commands.

There can be quite a few chars in the char queue, there can be a few completed command structs in the command queue.

The reason for using command structs is that by the time you get to the big case statement where you branch to the proper command handler code, you already know what case number to branch to.  String comparison was done when the struct was built.  There is a case selection number as one element of the struct plus some parameter fields.  These can be unions differentiated by the case selection number.

Brush up on queues, structs and unions.  It'll all become quite clear.
 

[NorthGuy]

The first question I'd be asking myself is whether there's a truly compelling reason to be using an 8-bit CPU at all, when there are so many more capable options out there. I started out learning microcontrollers using a PIC18, but now I use STM32 processors unless there's a compelling hardware reason not to (eg. unique peripherals, voltage range, temperature tolerance and the like).

To handle incoming serial commands, I first set up a receive buffer, and program a DMA controller in circular mode to capture all incoming serial traffic into it. This ensures that all received characters are stored without wasting time processing an interrupt for each and every one.

Then, at regular intervals, I check the DMA controller to see how many characters have been received since the last time round the loop. If there are any, I copy them into a separate buffer which holds a complete command line. This means the parser doesn't have to cope with wrap-around at the end of the DMA buffer. I could also take the opportunity to filter out invalid characters, and to ignore any line which contains them.

Finally, when a command is deemed to be completely received (ie. a CR/LF), the contents of the command buffer are parsed. Any remaining characters in the DMA buffer can simply be left there, as they'll be retrieved next time round the loop.

This method requires enough RAM to hold a complete maximum-length command line, plus as many characters as may ever be received in between times round the main loop - probably not that many in most cases, and you'd need somewhere to put received characters anyway when you're busy parsing a previous command. Major plus points are that there is no need for any code whatsoever to run each time a character comes in, and unless the receive buffer is too small, no incoming characters can ever get lost, however busy the CPU might be.

Downside? You can't do it on a CPU that lacks a DMA controller.

Wow!

I use ping-pong buffers for such tasks. The interrupt code fills in one linear buffer while the "main" code parses the other. When an end-of-line is detected, the interrupt switches to the other buffer. Very little code compared to what you're doing and still works well on PIC16.

[AndyC_772]

That's fine, provided you can be sure to process each command before the whole of the next one has been received.

The reason I like the DMA method is it completely eliminates any time dependencies. Provided the circular buffer is big enough, there's no requirement to ever complete action 'A' before event 'B' occurs, and this is useful when commands are coming thick and fast but some of them take time to execute.

Instead, the main loop just grabs as many characters as it can, or needs, from the DMA buffer, and parses them at its leisure.

It requires less code than you might think. Bear in mind, there's no receive ISR at all.

[NorthGuy]

The reason I like the DMA method is it completely eliminates any time dependencies. Provided the circular buffer is big enough, there's no requirement to ever complete action 'A' before event 'B' occurs, and this is useful when commands are coming thick and fast but some of them take time to execute.

Instead, the main loop just grabs as many characters as it can, or needs, from the DMA buffer, and parses them at its leisure.

This is an illusion. If the main loop can process X commands per second and you receive Y commands per second, a good design must ensure that X >= Y. Otherwise, no matter how big your buffers are, if the commands keep coming, your buffer will overflow.

Besides, I don't think you need to worry about the speed when you receive SMS messages from GSM modem. Just throttle the UART speed to the level where you can handle everything without problems and it'll be just fine.

[Dumble]

Hi all and thanks for replying. I've received far more help, replies and suggestions than I had anticipated getting so thank you all very much indeed. I've certainly got plenty to contemplate now

So I spent yesterday toying around with the example code I posted a link to yesterday and I managed to get that working and reading an SMS into an Array of Char. But then when I tried serial printing out the results to the serial monitor console, I found that I had only managed to capture the header data and not the SMS message body content.

After a short while spent head scratching, I finally realised that the SMS message header has a new line and or carriage return at the end of the header data before the SMS Message body and that's why the SMS message content wasn't getting read in. DOH!

Having thought about it, the fact that the two message parts are separated like that will be a benefit to sorting through the data.
I can store the first line of data (message header) into one array to find and validate the senders number and then store the message content into another array to find and sort the commands from the SMS Message body.


RE: Error Checking:-

The main reason I was going to include error checking was because during my testing, I've had times where the MCU has requested serial data from the GSM Module and I've had garbage data come back such as strange characters like odd letter Y and A - ALTHOUGH admittedly, this was probably due to errors in the code introduced by me like reading too much too quick etc. I just thought it might be beneficial to check for that happening and deal with it when the project is "out in the field" so to speak.

But if it's not necessary, I'm happy to leave that out.

Anyone can send an SMS to your device. And now, when you published the interface, anyone can control it.

I'm not quite sure what you mean NorthGuy?
You would have to first know the Mobile Number for the SIM Card used in the GSM Module to send it commands and before the system would accept and process the commands sent to it, the senders number would need to be validated against the one(s) hard coded into the software.

Or at least that's the plan of action

The first question I'd be asking myself is whether there's a truly compelling reason to be using an 8-bit CPU at all, when there are so many more capable options out there. I started out learning microcontrollers using a PIC18, but now I use STM32 processors unless there's a compelling hardware reason not to (eg. unique peripherals, voltage range, temperature tolerance and the like).

To handle incoming serial commands, I first set up a receive buffer, and program a DMA controller in circular mode to capture all incoming serial traffic into it. This ensures that all received characters are stored without wasting time processing an interrupt for each and every one.

Then, at regular intervals, I check the DMA controller to see how many characters have been received since the last time round the loop. If there are any, I copy them into a separate buffer which holds a complete command line. This means the parser doesn't have to cope with wrap-around at the end of the DMA buffer. I could also take the opportunity to filter out invalid characters, and to ignore any line which contains them.

Finally, when a command is deemed to be completely received (ie. a CR/LF), the contents of the command buffer are parsed. Any remaining characters in the DMA buffer can simply be left there, as they'll be retrieved next time round the loop.

This method requires enough RAM to hold a complete maximum-length command line, plus as many characters as may ever be received in between times round the main loop - probably not that many in most cases, and you'd need somewhere to put received characters anyway when you're busy parsing a previous command. Major plus points are that there is no need for any code whatsoever to run each time a character comes in, and unless the receive buffer is too small, no incoming characters can ever get lost, however busy the CPU might be.

Downside? You can't do it on a CPU that lacks a DMA controller.

Hi,

I guess the only reason I am using an 8-Bit MCU (Atmega328P-PU) is because that's what I'm used to working with and I guess you could say I'm in my comfort zone using the Atmegas. That and I've had no real experience working with anything other. A friend did try and persuade me to move over to PIC chips but I'm not at the kind of level where I can see the need to use a PIC over an Atmega just yet.

Having said that, one thing I can't help but think is that I've done myself no favors relying on the Arduino and its IDE.

And by that I mean I've allowed it to encourage me to be lazy with my programming through the luxury of endless libraries and pre-built functions and classes at my disposal. Whereas back in College, we were expected to write practically everything we needed for our programs, except for the functions already available in C, Arduino has it all there for you.

All you need to do is include a library, call the functions you want and away you go. After a while of doing that, you tend to forget how to do it yourself. If that makes sense?


Where I'm at with my Code at the Minute:-

To make life easier, I have taken the GSM module and have connected it up to an Arduino Nano so I don't have to deal with the other hardware in my project and I can just concentrate on writing the functions for the GSM in its own sketch. Once perfected, I will incorporate it into my main program

So yesterday, I spent some time tinkering away just seeing what was being read in and what was being printed out and trying to establish a flow of data. At the moment, the code contains a great deal of comments while I'm developing it. I tend to write notes down to remind me what direction I'm taking with the code at certain points, what certain parts of the code should do and all that sort of stuff. Just in case I have to stop working and can't get back to it for a few days.

Code: [Select]

//-------------------------------------------------------------------------------
//  READ, VALIDATE AND PROCESS SMS COMMANDS FROM SIM800L MODULE VIA SERIAL COMMS
//
//                 CAUTION: This Program is a Work in Progress!
//-------------------------------------------------------------------------------
//
//
//  LAST EDITED:- 16th Dec 2017 @ 09:58 Hrs
//
//  TURN OFF COMMAND ECHO WITH ATE0 AND THEN AT&W TO MAKE IT PERMANENT!
//
//-------------------------------------------------------------------------------

//  Using the Least Number of Libraries as Possible to Save Space / Resources
#include <SoftwareSerial.h>
SoftwareSerial MYGSM(2,3);      // Assign GSM Software Serial to Digital Pins 2 & 3 [RX/TX]

// SMS Packet Markers & Indicators
#define SOP '<'                 // Start of Packet Marker
#define EOP '>'                 // End of Packet Marker
#define SEP ';'                 // Separation Marker for Indicating Separation Between Multiple Commands

//  All Commands Have a Three Letter Prefix Followed Directly by Their Values. 
//  Example Command Packet Format:-
//  Multiple Commands: <MCT1;txF145.000;TAE0>
//  Single Command: <MCT1>


// Flags
bool started = false;           // Start Reading Serial Data (SMS)
bool ended = false;             // Finish Reading Serial Data (SMS)

// Data Variables
char inData[100];               // Array of Char Buffer for Holding Entire SMS Message inc Header Information (Sender No, Time & Date Stamp Etc.)
byte index;                     // Position Keeping Index for Buffer

//============

void setup() {
    Serial.begin(9600);         // Hardware Serial to Serial Monitor Begin at 9600 Baud
    MYGSM.begin(9600);          // GSM Software Serial Begin at 9600 Baud

    // Send a Quick Indicator Message Over Serial Monitor - [FOR TESTING ONLY]
    Serial.println(F("Starting..."));
    Serial.println(F(""));
     
}

//============

void loop()
{
    rxSMS();                   // Get the SMS Message
    delay(10000);              // Wait Some Time - Meant to Simulate the Periodic Passage of Time While the Atmega is in Low Power Sleep Mode - [FOR TESTING ONLY]   
}

// FUNCTION TO REQUEST AND GET THE SMS MESSAGE FROM THE SIM800L GSM MODULE
void rxSMS(){
   
    // Handshake Type Wake Up & Do Stuff Command...
    while(MYGSM.available() <= 0){

        Serial.println(F("HANDSHAKE BEGINNING..."));
        delay(2000);    // Quick Delay to Read The Serial Message
     
        // Set the GSM Module's SMS mode to ASCII
        MYGSM.println(F("AT+CMGF=1\r\n"));
        delay(50);
   
        // Set the SMS Notification Mode on the GSM Module - Some Modes PUSH the Message / Notification Straight Out Through Serial - We Don't Want This!
        MYGSM.println(F("AT+CNMI=2,1,0,0,0\r\n"));
        delay(50);

        // Request SMS Message from SIM Memory Storage Location #1 (Messages are deleted after use so new msgs should always be in location 1)
        MYGSM.println(F("AT+CMGR=1\r\n"));
        delay(50);  // DON'T LIKE THIS DELAY IDEA, IT CAUSES TROUBLE...
    }
   

    // Read all serial data available, as fast as possible
    while(MYGSM.available() > 0){
        char inChar = MYGSM.read();    // Circular Buffer? - Char inChar is recycled / used repeatedly to store each single character as it is read in from GSM Serial.
        if(MYGSM.available() > 0){

            // ALL SERIAL PRINT LINES ARE FOR TESTING ONLY!
            Serial.print(F("Char Read in: "));
            Serial.print(inChar);
            Serial.println(F(""));

            //----------------------------------------------------------------------------------------------------------------------------------------------
            //  CODE SHOULD GO HERE FOR FIRST READING THE SMS DATA INTO A CHAR ARRAY BUFFER,, THEN DO ERROR CHECKING OF THE DATA, THEN VALIDATE THE
            //  SENDER'S PHONE NUMBER AND FINALLY, IF ALL CONDITIONS ARE MET, EXTRACT THE PACKET DATA CONTAINING THE COMMAND(S) FROM THE SMS & PROCESS THEM
            //-----------------------------------------------------------------------------------------------------------------------------------------------
            //
            //1. READ ALL OF SMS INTO CHAR ARRAY (BUFFER) INC HEADER INFO SUCH AS TIME STAMP, SENDER NUMBER ETC.
            //2. CHECK SMS FOR ERRORS
            //3. EXTRACT AND VALIDATE SENDER'S NUMBER FROM SMS
            //4. PROCESS SMS TO FIND PACKET DATA (COMMANDS)
            //
            // NOTE... SMS MESSAGES APPEAR TO COME IN THE FORM OF HEADER DATA FIRST AND THEN A NEW LINE AND OR CARRIAGE RETURN FOLLWED BY THE MESSAGE DATA (PACKET).....
            // SCAN FOR NEW LINE OR CARRIAGE RETURN TO DETECT THE NEXT LINE BEING THE MESSAGE CONTENT OR PACKET!
           
             

            // This Part of the Code Looks for the Packet Start and Packet End Markers / Indicators From the Incoming Serial Data
            // THIS CODE SHOULD REALLY GO IN THE "PROCESS SMS" FUNCTION TO LOOK THROUGH THE DATA ALREADY IN THE BUFFER RATHER THAN READING IT AS IT COMES IN OVER SERIAL!
            if(inChar == SOP){
                 index = 0;
                 inData[index] = '\0';
                 started = true;
                 ended = false;
            }else if(inChar == EOP){
                 ended = true;
                 break;
            }
            // This part of the code takes the characters located between the Packet Start & End Markers and stores them in the inData array one by one followed by a NULL terminator.
            else{
                if(index < 79){
                  inData[index] = inChar;
                  index++;
                  inData[index] = '\0';
                }
           }
           
        } // End WHILE GSM Available
   
    } // End IF GSM Available
     
        // We are here either because all pending serial
        // data has been read OR because an end of
        // packet marker arrived. Which is it?
    if(started && ended){
     
        // The end of packet marker arrived. Process the packet
        // Do Something with the Packet

        processSMS();   // PROCESS THE SMS CONTENT
     
        // Reset for the next packet
        started = false;
        ended = false;
        index = 0;
        inData[index] = '\0';
    }
   
} // End rxSMS Function

// SMS DATA ERROR CHECKING FUNCTION - ONCE SMS MESSAGE HAS BEEN READ, CHECK IT FOR GARBAGE BY COMPARING IT AGAINST A SECOND READ OF THE SMS MESSAGE..
void smsErrorChk(){

 
}

// VALIDATE SENDER FUNCTION - EXTRACTS THE SMS SENDER'S NUMBER FROM THE SMS HEADER AND VALIDATES IT
void validateSender(){
 
}


// SORT, PROCESS AND DO STUFF WITH THE COMMANDS RECEIVED VIA THE SMS "PACKET"
void processSMS(){
   // if(ended == true)
   // {
        int i = 0;
        while(i < sizeof(inData)){
            Serial.print(inData[i]);
            i++;
        }
    //}
     
}

// DELETE ALL SMS MESSAGES STORED ON THE SIM CARD FUNCTION - NOT ENTIRELY CONVINCED OF THIS FUNCTION'S EFFECTIVENESS IN ITS CURRENT FORM...
//void delSMS(){
    //if(messageEnd == true){
       // MYGSM.println("AT+CMGD=1,4\r\n");
        //messageEnd = false;
       // Serial.println(F("Messages Deleted")); // [SERIAL CONSOLE MESSAGE FOR TESTING PURPOSES ONLY]
    //}
//}
 


So that's where I'm up to at the minute. Just plugging away at it, slowly and steadily.


Thanks again!

[AndyC_772]

This is an illusion. If the main loop can process X commands per second and you receive Y commands per second, a good design must ensure that X >= Y. Otherwise, no matter how big your buffers are, if the commands keep coming, your buffer will overflow.

Besides, I don't think you need to worry about the speed when you receive SMS messages from GSM modem. Just throttle the UART speed to the level where you can handle everything without problems and it'll be just fine.

Imagine for a moment that your device receives the following set of commands, one after the other:

1) Obtain a GPS fix
2) Report current location
3) Report battery level
4) Report state of logic inputs

How long does each command take to execute? What happens to each command string in the meantime?

[nctnico]

I solve these kinds of projects like this: have a circular buffer for the incoming characters which is long enough to contain at least one message. Usually I just use a length of 256 bytes. A UART interrupt fills the buffer and ignores the character which don't fit. In the main loop I have a command input buffer which gets filled by the characters from the circular UART receive buffer. However usually I only put characters which are text (>=32 and <=126) into the input buffer. If a character arrives which indicates an end of a line then I have the input buffer parsed (check for length, valid command and execute) and cleared. If the input buffer overflows I just empty it (appearantly receiving junk). This system doesn't guarantee every command gets processed but it does always recover from a fault situation or a malformed command. It also takes the realtime requirement out of the equation because receiving characters from the UART is running as a seperate process/thread.

[NorthGuy]

How long does each command take to execute? What happens to each command string in the meantime?

You just do the timing calculation. If you operate on string, you just figure out how much time you need to parse the longest string (which include initiating all the processes etc.). Say, it is 100 us.

This longest line may be followed by a shortest line. If you use ping-pong buffers, then you must complete the processing of your longest line while the shortest line is being received. Say, the shortest line is 3 characters (including LF at the end). You need to make sure it takes longer than 100 us to receive, which means 33 us per character or 3.3 us per bit - 300k baud. If you run the communication at 300k or slower, you won't have any problems. Say, 115200 will meet your timing with huge margin.

If you need faster than 300k baud, then the ping-pong buffer is obviously too slow, so you need to do more work to manufacture something faster. But there's a limit of what you can do. To find this limit, you find a string which is the slowest to process per character. Say, you found a 4-character string which takes 60 us to process, 15 us per character. If you get bombarded non-stop with such strings, one after another, you need to make sure that you can process data faster than you can receive. Which means maximum 15 us per character - 1.5 us per bit - 667k baud. If you go faster than that, say at 1M baud then you will receive more than you can chew. At 1M baud you can receive 100k characters every second, but you can process only 67k, so 33k characters will accumulate every second. Even if you have 100KB buffer, it will overflow in 3 seconds.

Now, the dilemma is whether I stay with my simple ping-pong code and accept the 300k baud limit, or do I go for more complex code to increase the limit to 667k. Or, perhaps I may work on my processing code to push the limit to 1M or more. It depends on the goal.

Of course, this doesn't apply to the OP application which is slow in the nature, so any MCU will easily keep up with the timing even if the approach is not efficient.

[AndyC_772]

I think you're confusing the time taken to receive a command with the time taken to process a command, which may not be trivial or deterministic.

Think about my example again. How long do you think it takes, or could take, to obtain a GPS fix?

How many characters can be received in that time? Where do you put them? Why do you think I'm recommending a method which doesn't try to split up the received character stream into separate commands until the parser is ready for them?

If it's the use of DMA you don't like, then bear in mind that you could easily use an ISR instead if your CPU doesn't have a DMA controller...

- character arrives from UART
- ISR puts it in a circular buffer
- foreground task periodically checks the buffer, and copies out any received characters up to the end of a complete command, leaving any others behind to be picked up next time round the loop
- parser executes command

[NorthGuy]

I think you're confusing the time taken to receive a command with the time taken to process a command, which may not be trivial or deterministic.

I do not confuse the time to receive with the time  to process. I compare them.

Think about my example again. How long do you think it takes, or could take, to obtain a GPS fix?

It doesn't matter. If you want to execute some lengthy process, you don't stop receiving and parsing until it's done. You merely indicate that the process must be performed. It doesn't take any significant time to set a flag requesting the process. Once the flag is set, the parsing continues. The process itself is going to be performed later.

If the execution of the command is simple and fast, the parser can execute it right away.

How many characters can be received in that time? Where do you put them?

You keep putting them in the ping-pong buffer when receiving, and you keep removing them from there when parsing/processing. This is a continuous process which doesn't stop to perform lengthy commands.

Why do you think I'm recommending a method which doesn't try to split up the received character stream into separate commands until the parser is ready for them?

Because you want linear execution - parsed, executed, done - pasred, executed, done - etc. You think that DMA gives you infinite time for the execution. It doesn't. If you get bombarded by the barrage of lengthy "fix" command, your buffers will overflow and the system will choke.

If it's the use of DMA you don't like, then bear in mind that you could easily use an ISR instead if your CPU doesn't have a DMA controller...

- character arrives from UART
- ISR puts it in a circular buffer
- foreground task periodically checks the buffer, and copies out any received characters up to the end of a complete command, leaving any others behind to be picked up next time round the loop
- parser executes command

You can do it this way, but you use double-buffering here - circular buffer for the input, then copying the characters into a linear buffer for parsing. It is much easier to use two ping-ponged linear buffers. The receiver fills one buffer, while parser reads from the other buffer. This removes the need for the intermediary copying, thus takes less memory and less code. Linear buffers are easier on the receiver too.