Program that can log/control many multimeters and other devices.

[KungFuJosh]

Per the subject: left TC running during the night, logging data from my S8000 battery charger. Woke up this morning and TC was initially responsive, managed to send a couple of commands to the charger and get its responses, but when I clicked on the "Chart" tab, it suddenly stopped responding and all I see is a blank window:

Check if there are new drivers available for your video card.

Thanks,
Josh

[KungFuJosh]

Question: Assuming SD in the grid panel readouts is Standard Deviation, what are the measurement units?

pico what? femto what? Certainly not Volts.

Thanks,
Josh

[dmenezes]

Check if there are new drivers available for your video card.

Just checked and there are no updates. Moreover, i don't think there's much chance of the problem being caused by a video card driver, as TC is pretty basic on its graphics (no 3D, no rendering, no shadowing, no animations, just very primary 2D usage on the chart) and also because everything else in the machine is working, including much heavier graphics programs (as a test, I just ran glxgears (which does all of these things) and it runs perfectly, while TC remains locked up.

Any more ideas? 

[KungFuJosh]

Check if there are new drivers available for your video card.

Just checked and there are no updates. Moreover, i don't think there's much chance of the problem being caused by a video card driver, as TC is pretty basic on its graphics (no 3D, no rendering, no shadowing, no animations, just very primary 2D usage on the chart) and also because everything else in the machine is working, including much heavier graphics programs (as a test, I just ran glxgears (which does all of these things) and it runs perfectly, while TC remains locked up.

Any more ideas?

An outdated graphics card driver will slow down minor things it should ignore, but that's not the case here.

What about Windows updates?

Is/was disk logging active in TC?

If you don't have a lot of memory, maybe the logging to memory in the table chart was the issue?

My system has 128GB of RAM, and the only time TC was an issue was an outdated graphics card driver. Though switching my meters to LXI instead of Socket has reduced the general responsiveness of TC. Mixed Socket + LXI was worse.

[dmenezes]

An outdated graphics card driver will slow down minor things it should ignore, but that's not the case here.

Agreed.

What about Windows updates?

As stated, I'm running it on Linux, not Windows -- and yes, Linux is fully updated.

Is/was disk logging active in TC?

No, logging to RAM. I have more than enough for what I'm logging.

If you don't have a lot of memory, maybe the logging to memory in the table chart was the issue?

As stated, at this exact moment (with TC hung) I have over 1.3GB of free RAM on the system and no swap activity, so I don't think memory could be the issue.

My system has 128GB of RAM, and the only time TC was an issue was an outdated graphics card driver. Though switching my meters to LXI instead of Socket has reduced the general responsiveness of TC. Mixed Socket + LXI was worse.

The problem here is not responsiveness: the machine is fully responsive and everything else runs great. The problem is TC being totally locked up, for hours, without even refreshing its screen.

[dmenezes]

The problem here is not responsiveness: the machine is fully responsive and everything else runs great. The problem is TC being totally locked up, for hours, without even refreshing its screen.

Trying to find what's going on, I just ran `strace -f` on the apparently hung TestController process; it's on a loop repeating the following non-stop:

Code: [Select]

strace: Process 14527 attached with 24 threads
[pid 16737] futex(0x7ff7c0004d60, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14553] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14549] futex(0x7ff7f02b617c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14548] futex(0x7ff7f0054130, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14545] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14544] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14543] futex(0x7ff7f0014988, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14541] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14540] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14539] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14538] futex(0x7ff7f00148d8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14536] futex(0x7ff7f00be978, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14535] futex(0x7ff7f0015db8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14537] futex(0x7ff7f0002720, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14534] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14532] futex(0x7ff7f0091840, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14542] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14531] futex(0x7ff7f005e100, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14530] futex(0x7ff7f0013878, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14529] futex(0x7ff7f0054130, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14528] futex(0x7ff7f0015608, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14546] futex(0x7ff7f0289d7c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14527] futex(0x7ff7f55df990, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 14528, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14533] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 14553] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out)
[pid 14553] futex(0x7ff7f031d328, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 14553] futex(0x7ff7f031d378, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=649744, tv_nsec=63660402}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14544] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out)
[pid 14544] futex(0x7ff7f00169d8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 14544] futex(0x7ff7f0016a28, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=649744, tv_nsec=79856977}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 14553] <... futex resumed>)        = -1 ETIMEDOUT (Connection timed out)
[pid 14553] futex(0x7ff7f031d328, FUTEX_WAKE_PRIVATE, 1) = 0


If it was a C program, I would be able to send it a SIGABRT to have it dump a core file, and then dig into it with a debugger to get the array with the collected data, and possibly even find out why it had locked up in the first place.

Is there any way to do anything similar with Java and/or TC?

[KungFuJosh]

Oops, missed the Linux thing. I'm mostly useless on Linux these days; it's been years since I had a Linux system running.

Hopefully, HKJ will have some insight there.

Thanks,
Josh

[dmenezes]

Hopefully, HKJ will have some insight there.

I held out for as long as I could, but @HKJ didn't respond -- no complaints, he already donates more of his time to this than anyone could reasonably hope for.

Today I tried to terminate the TC process (which is still in the same loop and still not even refreshing its window), and more weirdness happened:

1) first I tried terminating it by clicking on the "close window" button ("X" at the top right corner of any window); waited about a minute and nothing happened;

2) from the command-line, located the process PID and executed a `kill ` (graceful termination signal) on it: ditto, nothing happened.

3) finally, did a `kill -9` (forceful termination signal): this time it finished.

Hoping this could be useful for HKJ or someone to find out what happened and (more important) how to prevent it from happening again.

As for myself, I'm going back to redo a very long test run since all my collected data was lost -- and from now on I will only be capturing to disk.

[HKJ]

When you have some time, could you please test with 3 or 4 devices connected via LXI? TC gets laggy when they're all LXI, and I don't know if that's a TC thing, or something about my system setup that slows it down.

I don't have the option of using socket anymore. I've tried different ways, but if the Siglent and Keysight are both powered on, they both need to be LXI for things to work right. I have all my meters connected as LXI now, and that seems more stable than 2 or 3 as LXI and the rest as Socket.

I switched 4 bench DMM to LXI and have been running them for some time, I see no issues.
I have frequently used one on LXI and the rest on socket, that also works without problems.

[HKJ]

Hi,

Per the subject: left TC running for a few hours, logging data from my S8000 battery charger. When I came back to it a few hours later, TC was initially responsive, and I managed to send a couple of commands to the charger and get its responses, but when I clicked on the "Chart" tab, it suddenly stopped responding and all I see is a blank window

The OS seems OK, memory is not an issue (over 1.3GB of RAM free and no swap activity -- so the system is not thrashing or anything) and everything else but TC seems to be working fine. CPU is fine too, utilization is under 10%.

I've been using TC extensively in this same setup (Linux Debian Bookworm with openjdk-17-jre-headless), and this is the first time it shows anything like this.

I would really like not to lose the data I was capturing. Any hints? 

I do not have any idea why TC locked up. TC is multitasking and it is possible you hit a deadlock in the code, but I have never seen it. This kind of bug can be very hard to find and the log dump you posted do not help me.

[KungFuJosh]

I switched 4 bench DMM to LXI and have been running them for some time, I see no issues.
I have frequently used one on LXI and the rest on socket, that also works without problems.

Yes, it seems to run fine until you start the grid panel. Please try the same test, and open a 1x3 or 1x4 grid panel with advanced readouts. You might notice lag with the first readout added.

Thanks,
Josh

ETA: Did you see my other question?:

Question: Assuming SD in the grid panel readouts is Standard Deviation, what are the measurement units?

pico what? femto what? Certainly not Volts.

[HKJ]

I switched 4 bench DMM to LXI and have been running them for some time, I see no issues.
I have frequently used one on LXI and the rest on socket, that also works without problems.

Yes, it seems to run fine until you start the grid panel. Please try the same test, and open a 1x3 or 1x4 grid panel with advanced readouts. You might notice lag with the first readout added.

Running with all four meters having a advanced readout, no issues.

One detail: if you are logging the reading may be synchronized to the logging, but the sample rate can be different. If readout samples faster than reading it will simply reuse the last value.

ETA: Did you see my other question?:

Question: Assuming SD in the grid panel readouts is Standard Deviation, what are the measurement units?

pico what? femto what? Certainly not Volts.

SD is based on the volt readout, I do not know of the top of my head what unit it result in. A fast check with google makes me believe it is same unit, i.e. volt in this case.

[dmenezes]

I do not have any idea why TC locked up. TC is multitasking and it is possible you hit a deadlock in the code, but I have never seen it. This kind of bug can be very hard to find and the log dump you posted do not help me.

Thanks for having a look, and please accept my apologies for providing a useless log -- it was the only one I could think of.

Do you have any recommendations for me to try and provide more useful information (perhaps even a log) the next time TC locks up? eg, perhaps configuring JRE with some kind of debugging/trace/logging option, or running some special version of TC with that added in?

TIA!

[HKJ]

Thanks for having a look, and please accept my apologies for providing a useless log -- it was the only one I could think of.

If a java program crashes a core dump is made and that can be analyzed, but that was not the problem here.
There exist a debugger that can connect to a running program, but I generally do not use it. I do debugging directly in my development environment, it probably works the same way, but the UI is integrated.

Do you have any recommendations for me to try and provide more useful information (perhaps even a log) the next time TC locks up? eg, perhaps configuring JRE with some kind of debugging/trace/logging option, or running some special version of TC with that added in?

If you redirect and log the console output from the program, there may be some useful error information (TC dumps error messages to console output and then mostly ignores them). But if the error is a thread error that may not help, because no error may be generated.

[KungFuJosh]

ETA: Did you see my other question?:

Question: Assuming SD in the grid panel readouts is Standard Deviation, what are the measurement units?

pico what? femto what? Certainly not Volts.

SD is based on the volt readout, I do not know of the top of my head what unit it result in. A fast check with google makes me believe it is same unit, i.e. volt in this case.

The same unit type would make the most sense. However, the values shown imply there's an error in the calculation.

Thanks,
Josh

[dmenezes]

If a java program crashes a core dump is made and that can be analyzed, but that was not the problem here.

I'm running TC on Linux, and on Linux (true to its Unix inheritance) any process can be made to generate a core dump: it's enough to send it a SIGABRT signal. It will result in a core file, containing the whole of the program's memory plus registers and stack, being saved to the process' current directory.

Would that be useful to you?

There exist a debugger that can connect to a running program, but I generally do not use it. I do debugging directly in my development environment, it probably works the same way, but the UI is integrated.

About this standalone (not integrated) debugger: You mean some especialized Java tool, or something more generic like `gdb`? Can you please provide a reference?

If you redirect and log the console output from the program, there may be some useful error information (TC dumps error messages to console output and then mostly ignores them). But if the error is a thread error that may not help, because no error may be generated.

Thanks for mentioning that. I will make a point, from now on,  of always running your `tcrundebug` script (which redirects both stderr and stdout to the tc.log file) instead of `tcrun` as I've been doing.

But please answer my other two questions above, re: generating a core file, and about the debugger that can be connected to the process, so I can try and get you more information besides tc.log the next time the lockup happens.   

TIA!

[HKJ]

If a java program crashes a core dump is made and that can be analyzed, but that was not the problem here.

I'm running TC on Linux, and on Linux (true to its Unix inheritance) any process can be made to generate a core dump: it's enough to send it a SIGABRT signal. It will result in a core file, containing the whole of the program's memory plus registers and stack, being saved to the process' current directory.

Would that be useful to you?

No, it is the Java runtime that generates the dump and there are java tools to analyze it, a raw core dump would not have the required format.

There exist a debugger that can connect to a running program, but I generally do not use it. I do debugging directly in my development environment, it probably works the same way, but the UI is integrated.

About this standalone (not integrated) debugger: You mean some especialized Java tool, or something more generic like `gdb`? Can you please provide a reference?

Sorry, but it is a very long time since I have used anything but the development environment. There is usually a lot of tools included with jave, try checking them.

If you redirect and log the console output from the program, there may be some useful error information (TC dumps error messages to console output and then mostly ignores them). But if the error is a thread error that may not help, because no error may be generated.

Thanks for mentioning that. I will make a point, from now on,  of always running your `tcrundebug` script (which redirects both stderr and stdout to the tc.log file) instead of `tcrun` as I've been doing.

tcrundebug do more than redirect output, it also runs TC in debug mode and that means it will output all communications and a bit more to the log screen and console. Do not do that, it is for finding and fixing problems in definitions.

[HKJ]

The same unit type would make the most sense. However, the values shown imply there's an error in the calculation.

I did some verification when I programmed it.
StdDiv also exist as a Type on the Math page, i.e. you can select calculate StdDiv for a column, it uses the same math as the one in the advanced readout.

[KungFuJosh]

The same unit type would make the most sense. However, the values shown imply there's an error in the calculation.

I did some verification when I programmed it.
StdDiv also exist as a Type on the Math page, i.e. you can select calculate StdDiv for a column, it uses the same math as the one in the advanced readout.

The SD on the meters show accurately in low microvolts. TC showing SD as low picovolts and low femtovolts is not accurate. These meters would cost a lot more if it was true.

Thanks,
Josh

[HKJ]

The SD on the meters show accurately in low microvolts. TC showing SD as low picovolts and low femtovolts is not accurate. These meters would cost a lot more if it was true.

StdDiv is a specific formula you apply to a list of numbers and get a result, it is not directly depend on the actual resolution or precision of the meter, only on the numbers.
If the numbers do not vary StdDiv will be very low, even a 3½ digit meter can have a StdDiv well below pV, if the digits stays the same all the time.

[KungFuJosh]

The SD on the meters show accurately in low microvolts. TC showing SD as low picovolts and low femtovolts is not accurate. These meters would cost a lot more if it was true.

StdDiv is a specific formula you apply to a list of numbers and get a result, it is not directly depend on the actual resolution or precision of the meter, only on the numbers.
If the numbers do not vary StdDiv will be very low, even a 3½ digit meter can have a StdDiv well below pV, if the digits stays the same all the time.

So it's not a normal standard deviation then? Your StdDiv measurement is orders of magnitude off from the standard deviation of the actual measurement.

[2N3055]

Let me contribute just one point.
If we just look at the data plotted, and vertical scales for the plot, StDev cannot be in pV or fV.

You could get a 0V Stdev, if you took a very stable 3.5 digit meter that never changes last digit. But a plot from that meter would be completely flat line..

P-P values seem correct. Stdev is off by order of magnitude if you ask me. We know what StDev should look like because meters have internal stats and we know order of magnitude from there.
Why, I don't know. I know returned data is Ok because we can see it is.

[HKJ]

I will check it out and see if there is a bug in the calculations.

[2N3055]

I will check it out and see if there is a bug in the calculations.

Thank you for the effort and for the very nice software,  by the way...

[dmenezes]

No, it is the Java runtime that generates the dump and there are java tools to analyze it, a raw core dump would not have the required format.
(...)
Sorry, but it is a very long time since I have used anything but the development environment. There is usually a lot of tools included with jave, try checking them.

No problem, and thanks for both clarifications.

tcrundebug do more than redirect output, it also runs TC in debug mode and that means it will output all communications and a bit more to the log screen and console. Do not do that, it is for finding and fixing problems in definitions.

Are you sure about that? I ask because these are the contents of `tcrun` and `tcrundebug`:

Code: [Select]

==> tcrun <==
# Linux command line
nohup java -jar TestController.jar >/dev/null 2>&1 &

==> tcrundebug <==
# Linux command line
echo Saved debug output in tc.log
nohup java -jar TestController.jar >tc.log 2>&1 &

As you can see, both are absolutely identical, except for `tcrundebug` redirecting `stderr` and `stdio` to a file (`tc.log`) instead of discarding them (`/dev/null`), and echoing a simple message telling the user that.

AFAICS there's no straightforward way for `TestController.jar` to 'know' whether it's being started from `tcrun` or `tcrundebug` (it can't even check the parent process name, as it runs in the background and the parent finishes right after starting it), so its own behavior should be exactly the same no matter which of these two scripts it's started from.

Or am I missing something?