Using GCC, pointers seem to generate more efficient code than indexing on x86-64 (the difference is small, and I last checked this on GCC 4.9, so take it with a pinch of salt), so the structure I've sometimes used for file or socket input parsers/chunkers is
typedef struct {
unsigned char *next; /* First buffered unread character */
unsigned char *ends; /* End of buffered data */
unsigned char *data; /* Dynamically allocated buffer */
size_t size; /* Size of the dynamically allocated buffer */
int fd; /* POSIX.1 file or socket descriptor */
unsigned int errs; /* Error events, bitmask */
} inbuffer;
#define INBUFFER_INIT { NULL, NULL, NULL, 0, -1, 0 }
The buffer size is dynamically managed by an inbuffer_need(inbuffer *, size_t) function, which ensures that there are at least the specified number of bytes buffered, unless end-of-input is encountered (which is one of the events in the errs bitmask). The same function obviously reads from the input stream when necessary, and before reallocating the buffer, moves existing data so that next==data. This means that after consuming leading whitespace, the parser can do an inbuffer_need(&buf, MAX_TOKEN_LENGTH+1) call, and be assured that the entire token is in the buffer, starting at buf.next.
Data from the buffer is consumed using two helper functions, inbuffer_next(inbuffer *) and inbuffer_skip(inbuffer *, size_t):
static inline int inbuffer_next(inbuffer *ib)
{
if (!ib)
return -1;
else
if (ib->next < ib->ends)
return *(ib->next++);
else
return inbuffer_next_slow(ib);
}
static inline size_t inbuffer_skip(inbuffer *ib, size_t n)
{
if (!ib)
return 0;
else
if (ib->next + n <= ib->ends) {
ib->next += n;
return n;
} else
return inbuffer_skip_slow(ib, n);
}
These use two internal "slow" helper functions,
enum {
INBUFFER_EOF = 1<<0,
INBUFFER_ENOMEM = 1<<1,
};
static int inbuffer_next_slow(inbuffer *ib)
{
if (ib->errs) {
return -1;
}
if (ib->next < ib->ends) {
if (ib->next > ib->data) {
const size_t have = ib->ends - ib->data;
memmove(ib->data, ib->next, have);
ib->next = ib->data;
ib->ends = ib->data + have;
}
} else {
ib->next = ib->ends = ib->data;
}
if (ib->ends - ib->data >= ib->size) {
/* Omitted: increase ib->size, realloc ib->data. */
}
/* Omitted: Read up to (ib->data + ib->size - ib->ends) bytes,
increment ib->ends by the number of bytes read. */
/* If successful, return *(ib->next++), otherwise -1. */
}
static size_t inbuffer_skip_slow(inbuffer *ib, size_t n)
{
size_t skipped;
if (ib->ends > ib->next) {
skipped = ib->ends - ib->next;
ib->next = ib->ends = ib->data;
} else
skipped = 0;
/* Omitted: Skip up to (n - skipped) bytes of input. */
/* Return the actual number of bytes skipped;
this is either n, or smaller (in case of end of input). */
}
Depending on the C library version, this can be much faster than standard C fgetc() input.
When parsing binary data structures, like PNG chunks, you do a inbuffer_need(&png, 8); to ensure png.next points to the 4-byte big-endian length and 4-byte chunk type; then inbuffer_need(&png, 12 + length); to read the entire chunk in memory (assuming acceptable wrt. memory use). To move to the next chunk, you do inbuffer_skip(&png, 12 + length); I use accessor functions to obtain the 32-bit values from png.next + offset in big-endian byte order (cast and shift each unsigned char, then OR them togeter; usually generates pretty efficient code).
Funniest thing is, I still haven't found a really good way to make the reallocation and low-level read() chunk size policy "automatic". Such policies are hard! It is always a tradeoff between efficiency and excessive memory use. The st_blksize field in the struct stat for the descriptor is a good start, but sometimes you want to use minimum amount of memory even if it means more system calls (and thus slower program); sometimes you know the user will have lots of RAM available anyway for the data to be parsed and there being lots of it, it is better to waste some memory but be as fast as possible. I can't even leave the decision to the programmer (even if it is myself), because they cannot be arsed to think about how to find that out (from the user); much less implement e.g. suitable compile-time defaults and command-line override options. I am leaning towards adding minsize, maxsize, and a function pointer to a resize() size policy function to the structure, with compile-time defaults.