Linux:: how to synchronize the main with a thread in C

[0db]

hi guys
I am still studying Multithreading in C. I know how to synchronize two or more threads via Mutex and Conditions, but I don't know how to stop a thread until the main program has reached certain conditions.

Is there any full guide with examples?

[SiliconWizard]

Use semaphores.
You can then wait on a semaphore in a thread, and post it in another thread (including the main thread) to wake the thread up again and exit the wait.

[PlainName]

Is there any full guide with examples?

What OS are you using or thinking about? If it's FreeRTOS you should download the 'hands-on tutorial'.

Chapter 6.4 describes interrupt synchronization which should cover what you want to know. The pukka FreeRTOS API guide details the functions with examples, but this tutorial covers why you might want to use them.

Edit: just noticed this is the Linux topic. Duh! But the principle remains the same whatever you're using.

[0db]

Use semaphores.
You can then wait on a semaphore in a thread, and post it in another thread (including the main thread) to wake the thread up again and exit the wait.

You write "main thread", but in my case, I am in the int main() { ... } body of a C program and I need to wait until a thread, which will never terminate,  reaches a particular state.

Can I sill use semaphores, mutes, condition variables, &C in this case?

[0db]

Code: [Select]

// C program to demonstrate working of Semaphores
#include <stdio.h>
#include <pthread.h>
#include <semaphore.h>
#include <unistd.h>
 
sem_t mutex;
 
void* thread(void* arg)
{
    //wait
    sem_wait(&mutex);
    printf("\nEntered..\n");
 
    //critical section
    sleep(4);
     
    //signal
    printf("\nJust Exiting...\n");
    sem_post(&mutex);
}
 
 
int main()
{
    sem_init(&mutex, 0, 1);
    pthread_t t1,t2;
    pthread_create(&t1,NULL,thread,NULL);
    sleep(2);
    pthread_create(&t2,NULL,thread,NULL);
    pthread_join(t1,NULL);
    pthread_join(t2,NULL);
    sem_destroy(&mutex);
    return 0;
}


basically, I have 16 threads, and their synchronization looks like the above code. However, in main() {} I have this line

Code: [Select]

int main()
{...

            /*
             * wait untill tip has completed its process
             */
            wait_for_tip_completed(tip);


wait_for_tip_completed is an active waiting loop

Code: [Select]

void wait_for_tip_completed( ....

    while (1)
    {
        if (tip.completed) return;
    }   

This somehow works, but ... it doesn't look the right way.

[Karel]

http://man7.org/linux/man-pages/man3/pthread_join.3.html

[0db]

http://man7.org/linux/man-pages/man3/pthread_join.3.html

"pthread_join" is for terminated thread, mine never terminates.

[Karel]

If your worker thread never terminates, then you can use an (atomic) (global) variable (e.g. an int).
The worker thread writes the status of that thread to that global variable and the main thread reads it repeatedly to check the status.
Use sleep() or usleep() during the repeat loop in the main thread. As long as one thread only writes and only one thread reads from that
global variable, it 's safe.

Or use a pthread_mutex, they are designed for that.

[nctnico]

If your worker thread never terminates, then you can use an (atomic) (global) variable (e.g. an int).
The worker thread writes the status of that thread to that global variable and the main thread reads it repeatedly to check the status.
Use sleep() or usleep() during the repeat loop in the main thread. As long as one thread only writes and only one thread reads from that
global variable, it 's safe.

Or use a pthread_mutex, they are designed for that.

Reading a variable works but you have to poll it. If you use a semaphore you can wait until the semaphore gets signalled and continue immediately so performance is maximal.

[SiliconWizard]

Use semaphores.
You can then wait on a semaphore in a thread, and post it in another thread (including the main thread) to wake the thread up again and exit the wait.

You write "main thread", but in my case, I am in the int main() { ... } body of a C program and I need to wait until a thread, which will never terminate,  reaches a particular state.

Can I sill use semaphores, mutes, condition variables, &C in this case?

Of course. It's still running inside its own thread just like any other.

[Nominal Animal]

POSIX threads, my favourite!

Let's say you want the created threads to block at a specific point, until main thread has reached that point.  The simplest way to achieve this is to use a shared global mutex, that the main thread locks before creating the threads, and unlocks when it reaches the specified point.  The created threads simply grab that mutex and release it immediately.  If the main thread has not reached that point yet, then the threads will block (wait at that point, without consuming CPU time) until the main thread unlocks the thread.

Example:

Code: [Select]

#define _POSIX_C_SOURCE  200809L
#include <stdlib.h>
#include <inttypes.h>
#include <stdio.h>
#include <pthread.h>
#include <limits.h>
#include <string.h>
#include <time.h>
#include <errno.h>

#ifndef  NTHREADS
#define  NTHREADS  5
#endif

#ifndef  MSDELAY
#define  MSDELAY  500
#endif

static pthread_mutex_t  main_point = PTHREAD_MUTEX_INITIALIZER;


static void *worker(void *payload)
{
    const int        id = (intptr_t)payload;
    struct timespec  now;

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: Thread %d grabbing mutex.\n", (long)(now.tv_sec), now.tv_nsec, id);
    fflush(stdout);

    pthread_mutex_lock(&main_point);
    pthread_mutex_unlock(&main_point);

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: Thread %d continuing.\n", (long)(now.tv_sec), now.tv_nsec, id);
    fflush(stdout);

    return NULL;
}

static unsigned long  millisleep(unsigned long msecs)
{
    struct timespec  req, rem;

    req.tv_sec = msecs / 1000;
    req.tv_nsec = (msecs % 1000) * 1000000;

    if (nanosleep(&req, &rem) == -1 && errno == EINTR)
        return 1000 * (unsigned long)rem.tv_sec + rem.tv_nsec / 1000000;

    return 0;
}

int main(void)
{
    pthread_t        ptid[NTHREADS];
    pthread_attr_t   attrs;
    struct timespec  now;
    int              i, result;

    if (pthread_attr_init(&attrs)) {
        fprintf(stderr, "Cannot initialize pthread attributes: %s.\n", strerror(errno));
        return EXIT_FAILURE;
    }

    /* These worker threads do not need more than the minimum stack.
       Create an attribute to tell the C library about this setting. */
    pthread_attr_setstacksize(&attrs, PTHREAD_STACK_MIN);

    /* Lock the main_point mutex. */
    pthread_mutex_lock(&main_point);

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: Main thread started.\n", (long)(now.tv_sec), now.tv_nsec);
    fflush(stdout);

    for (i = 0; i < NTHREADS; i++) {

        result = pthread_create(&(ptid[i]), &attrs, worker, (void *)(intptr_t)(i + 1));
        if (result) {
            fprintf(stderr, "Failed to create thread %d of %d: %s.\n", i + 1, NTHREADS, strerror(errno));
            return EXIT_FAILURE;
        }

    }

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: Main thread sleeping for %d milliseconds.\n",
           (long)(now.tv_sec), now.tv_nsec, MSDELAY);
    fflush(stdout);

    millisleep(MSDELAY);

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: Main thread woken up; releasing mutex...\n", (long)(now.tv_sec), now.tv_nsec);
    fflush(stdout);

    pthread_mutex_unlock(&main_point);

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: Main thread mutex released.  Reaping worker threads...\n", (long)(now.tv_sec), now.tv_nsec);
    fflush(stdout);

    for (i = 0; i < NTHREADS; i++) {
        result = pthread_join(ptid[i], NULL);
        if (result) {
            fprintf(stderr, "Warning: Could not reap thread %d of %d: %s.\n", i + 1, NTHREADS, strerror(result));
        }
    }

    clock_gettime(CLOCK_REALTIME, &now);
    printf("%ld.%09ld: All done.\n", (long)(now.tv_sec), now.tv_nsec);
    fflush(stdout);

    return EXIT_SUCCESS;
}
Compile this using e.g.
    gcc -Wall -O2 -DNTHREADS=8 -DMSDELAY=250 example.c -lpthread -o example
and run using e.g.
    ./example | sort -n
The NTHREADS defines the number of threads, and MSDELAY how long the main thread sleeps after creating the threads.  The output timestamps are in seconds since Unix Epoch, real time.  The sort -n command sorts the output according to the timestamps.

On my system, this outputs

Code: [Select]

1585648512.613949989: Main thread started.
1585648512.614027646: Thread 1 grabbing mutex.
1585648512.614046279: Thread 2 grabbing mutex.
1585648512.614060944: Thread 3 grabbing mutex.
1585648512.614078104: Thread 4 grabbing mutex.
1585648512.614093435: Thread 5 grabbing mutex.
1585648512.614136953: Main thread sleeping for 250 milliseconds.
1585648512.614142279: Thread 6 grabbing mutex.
1585648512.614146957: Thread 7 grabbing mutex.
1585648512.614157421: Thread 8 grabbing mutex.
1585648512.864261296: Main thread woken up; releasing mutex...
1585648512.864304643: Main thread mutex released.  Reaping worker threads...
1585648512.864377347: Thread 1 continuing.
1585648512.864460867: Thread 2 continuing.
1585648512.864515911: Thread 3 continuing.
1585648512.864538436: Thread 4 continuing.
1585648512.864579944: Thread 5 continuing.
1585648512.864601765: Thread 6 continuing.
1585648512.864620224: Thread 8 continuing.
1585648512.864649748: Thread 7 continuing.
1585648512.864709077: All done.
which shows that all eight threads continued within 388 microseconds (0.000388 seconds) of the main thread releasing the mutex.

This works for situations where you have a "checkpoint" where you want threads to wait, until a specific thread lifts that "checkpoint".

[0db]

Perfect! 

[Nominal Animal]

Let me know if you want to see some other example, like thread pool/work pool, etc.  You could say I'm a POSIX C enthusiast.

In practice, I try to design my locking/synchronization schemes so that a given thread holds at most one lock at a time, to avoid the possibility of deadlocks.

When multiple concurrent locks need to be held (say, solving the dining philosophers problem), I write small test programs to stress-test the corner cases of my locking scheme.  These require a lot of work to design correctly, so looking at the literature and other resources to examine the patterns others have used, is a good idea.

[thinkfat]

For waiting for a particular state, you best use pthread conditional variables. IOW, pthread_cond_wait() and pthread_cond_broadcast() combined with a mutex.

[Nominal Animal]

Condition variables can be thought of as a parking lot, with the associated mutex being the single entry-exit gate or bottleneck -- the kind you need to grab/own to get through.

In my example, I only have the gate mutex.  Because the other threads pile up without issues while the main thread holds the gate, there is no need for a condition variable.

A thread pool or producer-consumer example would show how condition variables are used in practice.  Typically, you have a mutex, a chain (linked list) or queue or heap or array of pending work, and one or two condition variables.  One condition variable is signaled on whenever a new work item is added (and whenever a thread that grabbed the next work item notices there is more work left).  If the number of work items is fixed/limited, then the second condition variable is signaled or broadcast on, whenever work items are completed.

In my experience, when there is some variable that reflects the state somehow, protected by a mutex, you end up needing one or more condition variables, for threads to efficiently (without wasting CPU time needlessly) wait for a specific state or just for state changes; with those who change the variable signaling or broadcasting on the corresponding condition variable.
When there is no such variable, as in my example, a simple mutex or semaphore tends to suffice.
Consider that a practical result of the KISS principle, and not a computer science axiom!

[0db]

Let me know if you want to see some other example, like thread pool/work pool, etc.  You could say I'm a POSIX C enthusiast.

Thank you, you are very kind. I have already applied your tip to my code, and it works marvelously, but it also would be great to see more examples.

I am developing a straight packet sniffer like tcpdump, with a couple of additional threads that also implement packet logger since it is useful for network traffic debugging.

[Nominal Animal]

I am developing a straight packet sniffer like tcpdump, with a couple of additional threads that also implement packet logger since it is useful for network traffic debugging.

That sounds very much like a producer-consumer setup.

The main problem I've found in such cases is handling memory use sensibly.  Logging a packet can take a relatively long time, and they tend to occur in bursts, so you do want to use a buffer to cache them.

Each IP packet can be up to 65535 bytes long, but typically they are much shorter.  Depending on your read/receive mechanism, you almost always will do a memory move from the receive buffer to a queue item structure (that will include additional fields, CLOCK_REALTIME timestamp at minimum).  In some cases, you have a classifier thread, that examines each received packet, and either discards them, or forwards them to a processing queue/heap.  (A heap is often useful if the packets have different logging/processing priorities; a priority or arrival-time min-heap is surprisingly powerful for this.)

The memory problem is that since some of the packets need to be kept longer than others, you get severe memory fragmentation.  (In particular, it is important to remember that in most operating systems, chunks of memory of up to say 120k, allocated via malloc(), are not returned to the OS when free()d, just kept in the allocation pool for that particular process.  This excarberates the memory fragmentation and memory use for a logger that just mallocs and frees each packet.)

There are several approaches used to mitigate this effect.  You can use memory "caches" or "slabs", that contain fixed-size units.  For example, you could have 1.5k (covering typical Ethernet 1500-byte payload packets), 3k, 6k, and so on, with earliest unused unit in each slab used first, with a separate index-to-slab slot mapping, so that an unit can be moved within the slab if necessary to significantly shrink it.  (Yes, to access a slab, you'll end up dereferencing a "pointer" twice, but that cost is acceptable considering the memory use efficiency.  IIRC, I first saw this in classic Mac OS, version 7.5, I believe.)

(Many developers have powerful workstations with SSD storage, and often miss how their applications behave on a more typical machine, especially one with a spinny-disk HDD; and their application can become a swap-happy memory hog if nothing happens for long enough to the HDD to spin down, and then a stream of logged packets come in -- and due to the memory fragmentation, the OS cannot keep up with the memory use and I/O load at the same time, bogging everything down, and in the worst case, invokes the OOM killer... Yuk.)

Compared to the memory management details (to make the logger truly robust), the threads are almost trivial!

If you are implementing a transparent filter/logger, say on a small Linux SBC with two 1GbE interfaces, then surprisingly the memory management becomes simpler, as the underlying protocol is allowed to drop packets due to congestion.  (When logging, you really should try to avoid that happening.)  Such bridge gadgets are especially useful in analyzing exactly what internet traffic some devices have, and also to act as hyper-paranoid firewalls/loggers for experimental networks (say, isolating a subnet with devices that use customized IP stacks, or IP stacks you developed yourself).

This is exactly the kind of task where software development really becomes software engineering.  You cannot rely on standards and assumptions on what "should" happen or almost always happens; it needs to be careful and robust.  For debugging reasons, you need to check for errors even if those errors rarely occur in practice; knowing where reality first differed from expectations when things go b0rk is important. Otherwise, your service is like a voltmeter that draws a significant current from what it is measuring, affecting the system being logged so badly that the results are completely irrelevant.

On the other hand, this is an excellent opportunity to exercise proper engineering principles, from source control (are you using Git yet?), to unit testing.  For example, when you implement the packet management stuff (as header and source files), you can write a test program that stress-tests it using generated packets, and measures the memory use/fragmentation, and speed.  If you write a new one, you can use that test program to verify the new one is actually better.  Similarly for the thread management and other stuff.

Do not forget to keep your documentation up to date!  In the code, the comments should describe your intent, the overall purpose of the code, and not what the code does.  We can read the code, so we can tell what it does; but we humans won't know why, unless the developer explained the reasons or described the algorithm in comments.

Again, if you run into problems or out of ideas on how to implement some specific scenario, do ping me; I'd be happy to help anyone interested in writing robust POSIX C code.

[magic]

The main thread is a thread like any other and all pthread functions work in it.

I also recommend having a look at the _timedwait versions of pthread wait functions, this can be handy when things go wrong

And newest versions of C and C++ have built-in standard threading functions, which work on all platforms.

[guenthert]

The main thread is a thread like any other and all pthread functions work in it.
[..]

  That isn't quite the case.  Ok, this is perhaps nitpicking, but there are subtle differences between the main thread and others in Linux, most significant perhaps being that the process terminates, if the main thread does.

[Nominal Animal]

That isn't quite the case.  Ok, this is perhaps nitpicking, but there are subtle differences between the main thread and others in Linux, most significant perhaps being that the process terminates, if the main thread does.

Actually, it is even more complicated than that.  When the main thread exits in Linux, the C library calls the exit_group syscall, which causes the process to terminate.  If the main thread were to exit calling the exit syscall, the other threads would continue as normal.  The kernel only terminates the process (in the sense that a parent can reap its exit status) after the final thread in the thread group exits (or any thread calls the exit_group syscall, or the process is terminated by a signal).

So, strictly speaking, it is the C library that treats the thread that runs main() specially in Linux; to the kernel, it is just one thread in the thread group.

This is not the only Linux kernel wrinkle the C library hides.  For example, the Linux kernel treats credentials (user and group) as per-thread attributes, not per-process as POSIX states.  When one calls setuid(), setgid(), setresuid(), setresgid(), etc., the C library actually uses an internal real-time signal (via rt_tgsigqueueinfo()) to each thread in the thread group to synchronize them.

(Edited to add:  So, if you ever write code where one thread changes the user/group, and releases say a mutex or signals on a condition variable to let another thread go on with some input/output thing that needs the new credentials, funnn-ky race conditions await.  These are rare things, however, and at least us oldtimers keep a very suspicious eye on those, because bugs in that kind of code has bitten us before: it once lead to an exploitable root hole via FTP, for example.  Just had to send a Ctrl+C at the right moment, and you got root on the server.  But anything fiddling with capabilities or credentials needs several suspicious pairs of eyes look-through it, anyway.)

Unless you play with seccomp(), or write freestanding C to run directly under the Linux kernel without the standard C library, this does not matter; and it is more practical to just think of the main thread as "slightly special", yes.

[SiliconWizard]

Unless you play with seccomp(), or write freestanding C to run directly under the Linux kernel without the standard C library, this does not matter; and it is more practical to just think of the main thread as "slightly special", yes.

Indeed. For most practical matters, it doesn't make much difference. Obviously there are some things you can't do with the "main thread", such as "join" it - it wouldn't make sense.