Siglent SDS1104X-E and SDS1204X-E Mixed Signal Oscilloscopes

[rf-loop]

Your trigger hysteresis is critical. I do not know your signal more but normally this kind of level adjustment is not wise. Of course it goes there below hyst bottom but in practice if natural signal have even some drift in offset it may stop working. Normallu it is wise to keep bit more room for hysteresis window. Just for better practice...

Trigger hysteresis specifications, 0.3div Trigger Noise reject OFF (default) and 0.8div Trigger Noise Reject ON.
If signal do not go under hysteresis bottom level before it break hysteresis top level it do not trig.

[HendriXML]

The samples/s don't seem to influence the failure rate in my situation. But then again I don't have random noise, but "engineerd wide width noise".

I upped the low pass a bit.

[HendriXML]

I tried to fetch faillures using 1993 segments, but then they dont' come up. Important info as well..

I will stop experimenting for now and sit on my thinking stone...

[HendriXML]

So when it's displaying waveforms realtime the faillure rate is 30%-50%.
If it is not displaying waveforms realtime it is 0%.

The displaying of wavefroms is cpu stuff. So I still guess there might be 2 separated pieces of logic handling the trigger. The "cpu" one being faulty.

It could also be that the problem is caused by the cpu not being fast enough under certain conditions. There's not much samples/screen but I work remotely.

That might be important info as well.

[rf-loop]

I tried to fetch faillures using 1993 segments, but then they dont' come up. Important info as well..

I will stop experimenting for now and sit on my thinking stone...

It is important finding. I think really importat at this time...


When you use sequence do you know how many it have dropped out.

You can test it perfectly.

Because your signal is made by generator.

Now if you can set it generate example 1000 wfm burst or near what ever number of segments scope is capable with this setting.. Set it generate exactly 1000 wfm burst.
Then set it for manual trig (manual start burst)
Do not start.
Set scope ready for just this sequence acquisition and let it go (start single sequence). Now it waits signal. start your burst.
After done look every 1000 segment is there. If there is not failed trig position as previously and also if it have captured all 1000 you can repeat and repeat it intil it is clear it do not loose any and not fail trig any.
Now you can also do so that sequence come ready after all segments get. Set generator for 100 burst and set sequence for 1000 segments. It come ready when it get all 1000 and in final it stop and display all segments overlaid on screen until you touch your scope settings. From overlaid image you can immediately see if there is any wrong positioned segment.
Of course in these cases scope mode need be for single sequence.

[HendriXML]

I'll next test with burstmode. (Might take a while)
But the first test I'll be doing is spacing the signal. My prediction is that no failures will happen then. I don't think it is primarely about time between trigger signals, but about time between cpu intensive processes. When given enough time for the cpu to display the wave until the next trigger signal comes it won't fail.
That would be my hypothesis.
Why? It doesn't occur when using background acquisition of segment mode. It occur very frequently when using awg signals, so there's maximum pressure on the bug, but It doesn't occur always.
If this can be proven, we're not there yet. It just a condition.  The question then remains why the trigger logic fails, it not randomly showing false trigger positions.
Getting to know more about that new awg signals should be crafted, which are simplified to the max.

[HendriXML]

When the signals are well spaced no failures are detected.

The faillures start at more than 12 waves/s. I know the screenshots says 15 Hz, but the awg does 12 waves/s

When using a wave of 17 ms duration, there is nothing special about 76 ms separation vs less. (Not a complete lapse on a circulair buffer for instance)

My guess would then be that it takes about 76 ms to process the waveform, and during that time it is sensitive to the trigger bug.

[kcbrown]

Just try to get evidence with this kind of other settings that BW On reject or stop wrong time captures.
Also please test overall can you find any effect using Trigger  Noise Reject ON.  I think previously you tried an no effect. But now you have more experiment with this fail so it is god to ask again.

Previously I tried trigger noise reject on and that did have an effect, but it didn't prevent the issue entirely, at least as I recall.

I'm running another test now with trigger noise reject enabled, and we'll see how that works out.

After that, I'll run the 20 MHz bandwidth limiter test (with trigger noise reject disabled).   Last time I tried that, I wasn't able to reproduce the issue, but it's always possible that I didn't give it enough time.  This time I'll give each test something like 12 hours or more.

That said, it looks like HendriXML has managed to reproduce the issue using an AWG signal, and that's really excellent, because it means we can give Siglent a fairly foolproof way to reproduce the issue themselves and, thus, track it down and fix it.

I really hope Siglent takes this seriously (the only reason I might believe they wouldn't is that there is a UI hang bug that I've had extensive discussions with Tautech about, and Siglent confirmed they were able to reproduce it.  And yet, quite some time after that, they still haven't released a firmware upgrade with the fix to it).

[HendriXML]

I'm done with experimenting because I'm confident enough about the fehler deutung / bug interpretation.

When the FPGA is processing ADC values it keeps track of when the low and high values are passed. Maybe in a certain way with hysteresis and right slope. The last passage (time position) is stored in a register.
When a trigger condition has been met,  it send a notification to the cpu. (Interrupt) . The cpu process must read the register as soon as possible, because if its too late, (because cpu was busy) that register can be overwritten with another pass. Not satisfying full trigger criteria, but satisfying a pass. This is than falsely seen as the trigger "time position".
I think this also might be the reason to have a hysteresis in the first place. Otherwise because of noise there could be many passes, before the Interrupt was handled.
Btw is can also be that the FPGA doesn't care about triggers at all, only about predefined passes. And that each pass is an interrupt on its own. This is a way of low level vs high level events I spoke off some posts before.
When using scpi triggerlevels and slope are attributes of a channel. This might explain why. (Not "grouped" by usage, but grouped by underlying technology)

[HendriXML]

The following could also be observed:
Without using pass/fail masks, but with infinite persistence failures can also be seen. But the time between waves can be much shorter (like 30 ms). The reason for this might be that managing masks is more cpu intensive.
Also: the valid trigger condition and the invalid one don't need to be both on screen simultaneously.

[HendriXML]

That said, it looks like HendriXML has managed to reproduce the issue using an AWG signal, and that's really excellent, because it means we can give Siglent a fairly foolproof way to reproduce the issue themselves and, thus, track it down and fix it.

My guess would be that they already know of it.
The problem in the bug interpretation I gave is that it is impossible to transfer any data at the exact moment the interrupt fires. And the FPGA may not be halted. So no foolproof synchronization is possible.  There is always a time window in which things can already be changed.
I must say the tested time window (between two high passes) is massively large..
Are interrupts delayed too long, or do they wait too long to read registers that might change?
It is also not a real-time operating system isn't it? (Linux?) So maybe they have not even full control over it. (A specific device driver should be used then? Reading registers of the FPGA asap)

[HendriXML]

Btw the FPGA is not allowed to be halted (a way of ensuring foolproof synchronization), but the register could have "a do not overwrite" status before triggering the cpu.
That status should then be cleared by the cpu. It should even be possible have that status reset on a read from the cpu. (A bit like a destructive read)
It could use a fifobuffer/queue.

I don't know whats possible within a FPGA, but that is not too much to be requesting of it?

[rf-loop]

The following could also be observed:
Without using pass/fail masks, but with infinite persistence failures can also be seen. But the time between waves can be much shorter (like 30 ms). The reason for this might be that managing masks is more cpu intensive.
Also: the valid trigger condition and the invalid one don't need to be both on screen simultaneously.

As far as I know, and I know least "some things".
SDS1000X-E have as they call it "hardware based" mask test. You can also make mask using computer but it is bit tricky because there is not nice mask editor tool for it and you need first find how it works. I have made these long time ago using hexedit for some previous models. I have some kind of documents even in some deep storage what I do not have here.
Also it produce full speed Pass/Fail output (same BNC what have trig out). It is not like some toys.
When normally run mask test it do not reduce wfm/s speed what is without mask test with current settings.
(if things have not changed (FW/HW) after I have made lot of performance tests when SDS1000X-E model was quite young. )

[kcbrown]

The issue *does* reproduce with trigger noise rejection enabled, but it takes quite a bit longer.  In this case, it took 45452 mask tests before it reproduced, with an average trigger rate of 24 Hz or less:



Here's the data for it (a 7-zip archive with the CSV, the binary, the setup XML, and the above screenshot): https://app.box.com/s/5zmf6xr4t8xypjjmqmw46oj574r6sab3

With respect to theories like interrupt handling and such, the plain fact here is that the amount of time between events is orders of magnitude longer than what the triggering system is capable of handling.  We're talking about a 60 Hz waveform here!   If there's a problem with the CPU getting around to handling the trigger conditions fast enough with a 60 Hz waveform, then we'd have every reason to believe that the trigger system would simply fall over with MHz-frequency waveforms.  No, I don't think that's likely at all.  And that's especially true since this issue reproduces even with trigger noise rejection enabled.

Instead, I'm pretty sure what we're up against here is either a race condition or a failure to properly and consistently impose the trigger conditions.  But only the guys at Siglent can truly say.


I'm now in the process of running the test with the 20MHz bandwidth filter enabled on the channel.  I'll give it at least 12 hours to reproduce.

[rf-loop]

The issue *does* reproduce with trigger noise rejection enabled, but it takes quite a bit longer.  In this case, it took 45452 mask tests before it reproduced, with an average trigger rate of 24 Hz or less:



Here's the data for it (a 7-zip archive with the CSV, the binary, the setup XML, and the above screenshot): https://app.box.com/s/5zmf6xr4t8xypjjmqmw46oj574r6sab3

With respect to theories like interrupt handling and such, the plain fact here is that the amount of time between events is orders of magnitude longer than what the triggering system is capable of handling.  We're talking about a 60 Hz waveform here!   If there's a problem with the CPU getting around to handling the trigger conditions fast enough with a 60 Hz waveform, then we'd have every reason to believe that the trigger system would simply fall over with MHz-frequency waveforms.  No, I don't think that's likely at all.  And that's especially true since this issue reproduces even with trigger noise rejection enabled.

Instead, I'm pretty sure what we're up against here is either a race condition or a failure to properly and consistently impose the trigger conditions.  But only the guys at Siglent can truly say.


I'm now in the process of running the test with the 20MHz bandwidth filter enabled on the channel.  I'll give it at least 12 hours to reproduce.

Even when this is not reason but... as can see making tests is complex. When you change hysteresis you need also care that your signal  trigger thresholds have enough room for work robust even when small offset drift.. Hysteresis is 0.8 div when NR is ON. I have looked your previous images and your signal bottom is very close this 3.5 - 0.8V.
Small thing but it is good practice to care all settings are compatible when change something.
But overall, good findings.  Hope later we know how BW affect.

Least I thing 0.8V hysteresis is bit too borderline.


Then there are these peaks. These must not affect but fun peaks still... in interesting position.

[HendriXML]

With respect to theories like interrupt handling and such, the plain fact here is that the amount of time between events is orders of magnitude longer than what the triggering system is capable of handling.  We're talking about a 60 Hz waveform here!   If there's a problem with the CPU getting around to handling the trigger conditions fast enough with a 60 Hz waveform, then we'd have every reason to believe that the trigger system would simply fall over with MHz-frequency waveforms.  No, I don't think that's likely at all.  And that's especially true since this issue reproduces even with trigger noise rejection enabled.

We could create a signal as the one I sketched. Which may give an insight what happens when times are less. (It should not be about period, but time between high passes)

I think the basis of the problem is what I have described. That it doesn't happen on other signals might be due to (unknown) methods to prevent this from normally happening (stuff like hysteresis and ..). But this one slides trough anyhow.
At this time one would normally stop spending time looking at the blackbox (not for all of us), but open it and look inside.

[HendriXML]

I think the basis of the problem is what I have described. That it doesn't happen on other signals might be due to (unknown) methods to prevent this from normally happening (stuff like hysteresis and ..).

What about some holdoff time.. ensured by the FPGA?
It is a bit like having a brid shit falling on your head. What were the chances that that would happen on that exact moment and place? Very slim, but when it happens you know a bird did it. 

[2N3055]

Just a comment, to me that time interval looks awfully close to limit..
I'm not sure signal climbed those two voltages in 1,2-1,4 ms, it could be less than 1,2ms..
Maybe vary that a bit to find out if that influences something to try to find more clues....

Also I find it confusing that L1 is higher than L2 (maybe better naming would be UL-upper limit and LL-lower limit)

[rf-loop]

Just a comment, to me that time interval looks awfully close to limit..
I'm not sure signal climbed those two voltages in 1,2-1,4 ms, it could be less than 1,2ms..
Maybe vary that a bit to find out if that influences something to try to find more clues....

Also I find it confusing that L1 is higher than L2 (maybe better naming would be UL-upper limit and LL-lower limit)

Have you read everything said in « Reply #1772 on: December 07, 2020, 07:34:46 am » and after it all what handle this trigger engine bug.

[2N3055]

Just a comment, to me that time interval looks awfully close to limit..
I'm not sure signal climbed those two voltages in 1,2-1,4 ms, it could be less than 1,2ms..
Maybe vary that a bit to find out if that influences something to try to find more clues....

Also I find it confusing that L1 is higher than L2 (maybe better naming would be UL-upper limit and LL-lower limit)

Have you read everything said in « Reply #1772 on: December 07, 2020, 07:34:46 am » and after it all what handle this trigger engine bug.

No. I'm afraid I didn't. I just made a comment to this image...

I do understand there is a problem, I'm just saying that playing with time limits might reveal something else.

L1 and L2 comment is my comment to Siglent's choice in U/I naming.

[rf-loop]

Just a comment, to me that time interval looks awfully close to limit..
I'm not sure signal climbed those two voltages in 1,2-1,4 ms, it could be less than 1,2ms..
Maybe vary that a bit to find out if that influences something to try to find more clues....

Also I find it confusing that L1 is higher than L2 (maybe better naming would be UL-upper limit and LL-lower limit)

Have you read everything said in « Reply #1772 on: December 07, 2020, 07:34:46 am » and after it all what handle this trigger engine bug.

No. I'm afraid I didn't. I just made a comment to this image...

I do understand there is a problem, I'm just saying that playing with time limits might reveal something else.

L1 and L2 comment is my comment to Siglent's choice in U/I naming.

Afaik these limits have also tried change and what looks like do not affect.
Yes it is tight limit as I also tell in my earlier msg.

- - - - - - - - -

Imho, L1 and L2. I look these just opposite, very logical.

Number one is... who is top.

Not so seriously but... 
See the book index. Page 1 is at the top. See the results list of the running competitions. 1 is at the top. When prizes for competitions are awarded. 1 stands at the top. And so on.
That numbering is also culture and language independent. UL and LL are perhaps something extremely fun in some place ..

[kcbrown]

13 hours and 480000 mask test frames later, the issue still hasn't reproduced with the 20 MHz bandwidth limiter enabled on the channel.  I'll be happy to keep it going for another 12 hours, but it's certainly looking like the 20MHz bandwidth limiter eliminates the issue in my environment.  But that's in my environment.  It may be that noise at a lower frequency than 20 MHz would just as easily trigger the bug.  On that, I can't say.  But it would be an interesting test for HendriXML to perform with his setup, seeing how he's able to reproduce the issue more or less at will with an artificially-generated signal.

[HendriXML]

I've made an arbitrary wave much like __2.png

This wave gives a lot of failures as well. I've tried different frequencys (with corresponding slope times) like 1 Hz, 50 Hz, and 1 Khz.

All fail.

BW limit on gives also failures, but that is expected (no noise, but an engineerd signal). Didn't try it on all frequencies.

I observed how far the trigger timings was off, this varies between square waves at the begining until a few divs further. At 50 Hz and 1 Khz this is fast switching but looks the same. I must say I would have expected the high freqencies to go further (maybe I should have been more subtle in the increase. Will have a test at 100 Hz.)
1 Hz goes to slow to see patterns in how the trigger location shifts.

The 50 Hz pattern is sometimes animated: a fraction of second no failiures  and then 1/10 div, 2/10 div .. 2 div (very rapidly)
Each time the trigger is further in the wave, and then no faillure for a fraction of a sec and then it starts again.

Here the timebased element (other than the signal) is clearly visible.

The next session I'll experiment with time separation again, maybe some info can be extracted from looking at the patterns.

[rf-loop]

13 hours and 480000 mask test frames later, the issue still hasn't reproduced with the 20 MHz bandwidth limiter enabled on the channel.  I'll be happy to keep it going for another 12 hours, but it's certainly looking like the 20MHz bandwidth limiter eliminates the issue in my environment.  But that's in my environment.  It may be that noise at a lower frequency than 20 MHz would just as easily trigger the bug.  On that, I can't say.  But it would be an interesting test for HendriXML to perform with his setup, seeing how he's able to reproduce the issue more or less at will with an artificially-generated signal.

This is good finding.

[kcbrown]

21 hours and 800000 mask frames, and the issue still doesn't reproduce with the 20 MHz bandwidth filter enabled.  I'm going to call it at this point, and say that in my environment and with this signal, the issue does not occur with the 20 MHz bandwidth filter enabled.

And little wonder: the strongest noise in the signal is at 25MHz: