Pocket-Sized 6 GHz 1 TS/s ET Scope

[joeqsmith]

Sorry but I am still not understanding why the mismatch.  In this last example, you start at 10.5.  The software shows data 125 ps prior.   If I send the D command with a 10ns delay,  I assume the data the scope sends is at the delay I specify, not 125ps prior.  Your plot shows switch points between the sample times.  Again, not what I would expect.   In my software, I am just setting the delay and reading back the data for each discrete point in time.    If I program 10.0 as a start, it starts at 10.0.  If the scope is sending me data at some other time, I am not aware of it.   

The data returned always corresponds to the time requested. What we are talking about pertains only to the display of the data.

Suppose we take data from 10 ns to 11 ns in 0.1 ns steps. Conceptually, the data is plotted as shown in the attached image. To make the data visible in intensity-graded mode, we shade a finite width of the screen centered around each point in time.

This means that the start and end point of the time axis does not correspond to the first and last time in the data.

In your previous example where you show the plot using 10.5, 10.75, and 11.0 ns, you actually plot 4 data points, even though the software is showing only 3. 

If I use my software to collect data starting from 10.5n with a range of 0.5n and a 250ps resolution, I get a start of 10.5 and a stop of 11 with three data points.   This is what I would expect to see.   

***
 
So ... I looked at LabView's intensity graph and sure enough, it has the same problem as what you show.  I had not noticed it until I looked at the three sample example.   When I scaled the graph originally, I just used the two end points.  Looking at a lot of data, I did not see the error.   

[joeqsmith]

Repeating the random test, attached showing the commands sent just prior to the DSO going unresponsive.    Once it enters this mode, it will no longer respond to any commands.

We cannot reproduce a lockup by sending this exact sequence of commands. We have been performing similar fuzzing tests on several units in parallel (integrated 200 hours) with no lockup.
There may be some difference between the way we are performing these tests. We are sending between 1 and 100 commands, each containing between 1 and 10000 random bytes, and then reading the responses. If possible, it would be helpful to have a sequence of commands that reproduces the problem. We are looking for codepaths that could cause what you're seeing, but not being able to cause the problem makes debugging difficult.

If we are calling a command one packet of data,  I send them until I decide to stop it.   The packet size is between 1 and 12 bytes.  All packets are CR terminated (adding one additional byte).    There are no filters as to what the payload contains.   It's just random data of a random length. 

Sure, I can create a test file of commands for you like last time.  It seems easy enough to reproduce. 

[joeqsmith]

Power cycle the scope and send these commands.   Next run your software.  You should find that the software will connect to the scope.   The scope should not trigger no matter what settings you make.

Next, power cycle the scope.  Run your latest software then exit.  Send these command to the scope.  Run your software.  The scope should not be detected.

***
Note this is a CSV file.  The file type isn't supported for uploading.  Packets are CR terminated.

[joeqsmith]

Not that is will help your situation but with LabView, I just clip off the first and last half time and then process the X axis normally.   Nice thing about this approach besides aligning the screen to the requested start and stop locations, it's also simple (at least with LabView) to add. 

[SJL-Instruments]

Not that is will help your situation but with LabView, I just clip off the first and last half time and then process the X axis normally.   Nice thing about this approach besides aligning the screen to the requested start and stop locations, it's also simple (at least with LabView) to add. 

Thanks for the suggestion. We will implement this behavior in the next revision (half-width shaded region on first and last points). You're not the first to ask about the axis labeling - hopefully this will prevent future confusion.

Power cycle the scope and send these commands.   Next run your software.  You should find that the software will connect to the scope.   The scope should not trigger no matter what settings you make.

Next, power cycle the scope.  Run your latest software then exit.  Send these command to the scope.  Run your software.  The scope should not be detected.

We have tried sending these commands in several configurations (in one large binary blob, or with commands separated by 0.1, 1, 10, 100, ms, both with or without reading the responses) but cannot induce a lockup.
We have attached a log from one of these tests if you'd like to verify that we are interpreting your file correctly, and splitting the commands as you'd expect. In this log, some commands take longer than the 10 ms timeout, and so the corresponding response appears after a later command (with the intermediate commands queued).
Based on your response #472, does the scope always lockup at the command "72 2D 72 0D"? If so, this may be enough information for us to track down the root cause in firmware (even if it is unit-dependent).

[joeqsmith]

I am not sure if it actually locks up.  It ends up in one of two modes, depending on if I had ran the new software or not.  One mode, it will not trigger after connecting to the scope with your latest software.  The other, it seems to go unresponsive and your software can not detect it on startup.  Windows still sees the connection.  In this state, I am able to use my software to connect to the scope but it will not respond to any commands.

Both cases, the scope will recover only after a power cycle. 

I just tried it again with the script I provided and was able to repeat the same results.   

Looking at your txt file, it appears correct.   

It appears that it is related to the new R command as symptom is tied to if I run your new software or not.   Currently I give the scope a ms to respond.  If it doesn't respond, I send the next packet.   This is the same way I tested the previous firmware.   I wonder if I am overflowing some internal buffer in your firmware.   I slowed down my transmission and it's up to 50k packets.   I'll let it continue to run but it is starting to look like an overflow problem.     

You mentioned you had looked at my VNA software.   If so, did you get the 32-bit application to run? 

Not that it helps, but on the graphing, LabView allows you to have multiple markers per axis on a graph.  The horizontal would normally be 0 - 10 for example, but I can have a second scale of say 10n - 10.5n on that same axis.   I can also manually scale the graphs.  So, I send the data to the graph and set the min and max horizontal to a half sample from the actual min/max.  I then set the second scale to what the actual time is.  There's no math or anything to track.  It's very clean to do it this way and I don't have a dead spot on the graph which is sounds like you will have based on your description.  If I tell the scope to sweep from 10-11, I am not expecting the graph to be from 9-12 with data only showing up between 10 and 11.  I expect it to start at 10 and stop at 11. 

***
Appears to be some type of overflow.  I suspect when I run your software and allow the scope to start to sweep, it places it into a mode where it is more susceptible to the overflow.   If you have LabView, I can give you the source code.  Or, if you have the runtime engine installed for 2011 (would have been required to run the original software I provided for the NanoVNA) I can supply you with an EXE.  My guess is there is nothing really unique about the data I am sending the scope, it's just how fast I am able to send it.  But you tried it with a 100us throttle, which should be faster than what I am doing.   

[SJL-Instruments]

So, we found the root cause of the issue.

First, we misread your reply #477, and thought that the commands themselves would cause a lockup. We can confirm that running the commands, in conjunction with our software, reproduces the behavior you're seeing. Sorry about that.

The main problem is that one of your commands sets the CDF tolerance (% command) to 0.000007, and our software does not reset it to the default (0.01) on startup. We will implement this in the next revision. This solves the "first lockup mode" you're describing, where the software connects but does not trigger.

The reason this causes a lockup is that the timeout for the R command is scaled based on the CDF tolerance (the expected amount of time needed to reach the specified tolerance), and we did not put a cap on this. In v15 firmware we will get rid of this, and allow the user to specify a max timeout manually, in seconds.

This change will solve the "second lockup mode" to some degree, but not completely. As an example, at very low trigger rates (100 Hz), each R command may need 5 seconds to gather enough data. It's possible to queue up hundreds of these read commands in the serial buffer, effectively "locking up" the scope for several minutes.

This behavior is technically "as designed," but the end result is not desirable. We could of course put a hard cap on the timeout, or decrease the buffer size, but this artificially limits the capability of the scope, and does not completely get rid of the issue. Another idea is to limit the total "queued time" of buffered commands, but the time taken is data-dependent and not predictable. If you have a preferred way to solve this issue, let us know.

On another note, this behavior is unchanged from v13. We did not find it, since our tests did not try fuzzing in conjunction with running the software. We will make this mode part of our testing in the future. The necessary conditions are for the CDF tolerance to be very low (<10 ppm), and the max samples to be very high (>3 million), which is difficult to find with random bytes.

We did check the firmware for any buffer overflow issues, and did not find any. Any commands that would overflow the buffer will be dropped, but should not cause a lockup.

Not that it helps, but on the graphing, LabView allows you to have multiple markers per axis on a graph.  The horizontal would normally be 0 - 10 for example, but I can have a second scale of say 10n - 10.5n on that same axis.   I can also manually scale the graphs.  So, I send the data to the graph and set the min and max horizontal to a half sample from the actual min/max.  I then set the second scale to what the actual time is.  There's no math or anything to track.  It's very clean to do it this way and I don't have a dead spot on the graph which is sounds like you will have based on your description.  If I tell the scope to sweep from 10-11, I am not expecting the graph to be from 9-12 with data only showing up between 10 and 11.  I expect it to start at 10 and stop at 11.   

We've attached a screenshot of the updated scheme - this should be identical to your proposal in #478.

[joeqsmith]

One possible solution would be to add a watchdog.  The software would send down a refresh command at some minimum rate.  The firmware would require this command, or any other communications at this minimum rate or it would return to the default power up state.  The firmware can't really count on the calibration command as the user may not want to send it.  The refresh command may not even have an acknowledge returned to the PC.

You may want some sort of fault status that the software could read to determine if a reset occurred.   

It's possible to queue up hundreds of these read commands in the serial buffer, effectively "locking up" the scope for several minutes.

That's fine as long as the PC would continue to send the refresh during this time.  If something went wrong and no data was being sent from the PC, the firmware would reset to the default state.   The timeout could be in the several seconds.  We are not trying to waste a lot of time with this sanity check.  We just want a way to recover if there is a major fault. 

I'm sure there are many ways to skin this one.   Give it some thought.

[SJL-Instruments]

After asking the good folks on the MCU forum, seems like the consensus, and industry-standard implementation, is to scrap the FIFO entirely (or reduce its length to 1). In our specific case, there is no extra throughput gain to be had by having a FIFO length of 2 or higher, and a FIFO length of 1 is sufficient if the end-user program can be mapped to a state machine. This avoids the lockup problem entirely.

There is a question of whether to actually remove the FIFO, or just remove it as part of the documented interface. Both options would behave identically as long as the user implementation follows the documentation (wait for response to start before sending next command). Removing it may break backwards compatibility - we are leaning towards keeping it for this reason. If it is kept, we would add a note in the documentation in case anybody does run into the lockup.

[joeqsmith]

I had changed to a full handshake some time ago and don't see a problem with it working as described.  As long as everything is documented, should be good to go.  I have added the tol command to my software as well and adding the Y to the R command. 

***

The reason this causes a lockup is that the timeout for the R command is scaled based on the CDF tolerance (the expected amount of time needed to reach the specified tolerance), and we did not put a cap on this. In v15 firmware we will get rid of this, and allow the user to specify a max timeout manually, in seconds.

Consider that what ever you come up with, there needs to be a way to abort a command (without pulling the USB cable).   As I mentioned, I use a full handshake today.  However, if the scope doesn't respond in N time, I do some sort of recovery.  Normally, resending the last command.  There isn't any mention in the manual how you want to handle such cases. 

[Kean]

There is a question of whether to actually remove the FIFO, or just remove it as part of the documented interface.

Are you able to detect software disconnection of the communication port to flush the FIFO.  e.g. via the CP2102 hardware handshaking signals or maybe the SUSPEND pins.
Not sure if SUSPEND is useful, as that probably cannot be triggered under app software control, and only by host power management.

[SJL-Instruments]

Consider that what ever you come up with, there needs to be a way to abort a command (without pulling the USB cable).   As I mentioned, I use a full handshake today.  However, if the scope doesn't respond in N time, I do some sort of recovery.  Normally, resending the last command.  There isn't any mention in the manual how you want to handle such cases. 

We could immediately cancel the current command, and flush the FIFO, when the Control-C (or Command+C) byte is received. This has the benefit of being natural for interactive terminal use. (We do not anticipate needing to send binary data in a command, other than during USB firmware update, where throughput is not a major concern.)

Are you able to detect software disconnection of the communication port to flush the FIFO.  e.g. via the CP2102 hardware handshaking signals or maybe the SUSPEND pins.
Not sure if SUSPEND is useful, as that probably cannot be triggered under app software control, and only by host power management.

For the reasons you mentioned, we don't think this would be a reliable mechanism for this use case. If a serial connection is closed by one program and opened by another, the SUSPEND pin will not necessarily go high in between, so the FIFO responses will continue to be processed and sent to the second program.
Our current plan is to nominally (in the documentation) require waiting for one response to start before sending the next command. We will keep the FIFO for backwards compatibility, but will mention the potential lockup issue if this guideline is not followed, as well as how to request immediate cancellation/flushing if needed.

[joeqsmith]

We could immediately cancel the current command, and flush the FIFO, when the Control-C (or Command+C) byte is received. This has the benefit of being natural for interactive terminal use. (We do not anticipate needing to send binary data in a command, other than during USB firmware update, where throughput is not a major concern.)

As I previously wrote, I have never understood your desire to use a terminal to control the scope.  You can't get any meaningful data without some sort of custom software.  As I also mentioned, I would rather have all communications processed the same way. 

https://www.eevblog.com/forum/testgear/pocket-sized-6-ghz-1-tss-et-scope/msg5327255/#msg5327255

If you do decide to implement some sort of abort command, make sure you think through it.  You already have no error checking in the protocol and two different terminators.  Using your company name as a terminator to work around binary data being sent....   Make sure that as you expand the protocol, you don't piecemeal it.

This is the reason I had mentioned the automotive standards.

[SJL-Instruments]

As I previously wrote, I have never understood your desire to use a terminal to control the scope.

We have one user who has requested a bulk voltage read mode (specify start/end/step and receive a voltage array), and would prefer using this from an interactive terminal. We do agree that for now, a custom program is needed to interpret the CDF data.

Although, on second thought, anybody using an interactive terminal would never build up a FIFO backlog anyway, so this concern is a moot point for the current discussion.

As I also mentioned, I would rather have all communications processed the same way. ...
Make sure that as you expand the protocol, you don't piecemeal it.

This is the reason I had mentioned the automotive standards.

We are working on implementing the SCPI standard for test equipment. Given that most of our planned changes for v15 concern the serial interface, we'll bite the bullet and commit to implementing this standard for v15 (rather than make piecemeal adjustments in between), even if it delays the update somewhat.

[joeqsmith]

As I previously wrote, I have never understood your desire to use a terminal to control the scope.

We have one user who has requested a bulk voltage read mode (specify start/end/step and receive a voltage array), and would prefer using this from an interactive terminal. We do agree that for now, a custom program is needed to interpret the CDF data.

Although, on second thought, anybody using an interactive terminal would never build up a FIFO backlog anyway, so this concern is a moot point for the current discussion.

That seems odd as the newer software allows ways to export the data.   Then there are other shortcomings, for example, they have something beyond a single state signal.  I could see having only the voltage could confuse them when taking measurements.   Guessing they understand the limitations of the scope and how it works.   IMO, the way it is today, if someone needed a feature beyond what the software support, I would suggest just asking you to add it.  Other's may be able to leverage these features.   If you really need something custom, there's no reason not to roll it from scratch.  It's not that complicated especially now with the improved documentation.     
 
If you continue to make changes to a protocol that you do not intend to support long term, you're just making work for yourself and anyone else who writes an interface for it.   This is why I wanted you do make sure you had a clear path moving forward early on. 

[SJL-Instruments]

We have just released v2.6.3 of the software and revision H16 of the manual. This fixes the bugs discussed here since the last update, adds tooltips (built-in help) for all controls, and FFT cursors. The manual now has a changelog, and we have added documentation on the MCU update protocol (per request).

We tried implementing Rj/Dj decomposition for Tj(BER) as well as mask testing for this update, but after several iterations we were not satisfied with the results. This is largely due to the single-comparator hardware limiting the BER fidelity to ~1e-4 in a reasonable acquisition timeframe. We did mention this in the manual, and made this clear to anyone asking about SI applications; but just in case, we've contacted all customers who mentioned SI, and extended their return period in case they were expecting this capability in the future.

A bit disappointing on our end, as we do want to push the hardware as far as it can go for the end-user. But best to be transparent about the limits of the hardware, once we reach them. More advanced jitter decomposition/BER analysis will require a future hardware model. Seems like the most successful applications for this scope as of now are high-speed timing analysis (setup/hold, rise/fall times, phase-matching) , TDR/TDT, qualitative eye diagram evaluation, and pulsed laser measurements.

[joeqsmith]

Tooltips is nice.  Consider providing a means to disable it and include that setting when saving the setup.   Bug wise, of course if you run that script I provided, and run your software without a power cycle, you still can't detect the scope is present.   I'm guessing you are aware of this and it will require a firmware change to solve it. 

It it much faster compared with when I first looked at it.  The whole user experience is much improved.  You've come a long way. 

I imagine if you turn the crank again (new products) it will get easier based on what you have learned.

[SJL-Instruments]

Tooltips is nice.  Consider providing a means to disable it and include that setting when saving the setup.   

Thanks - we will include this in the next update.

Bug wise, of course if you run that script I provided, and run your software without a power cycle, you still can't detect the scope is present.   I'm guessing you are aware of this and it will require a firmware change to solve it.

Yes, this will be fixed in the v15 firmware update. Currently we're estimating this to be released in mid-May. The planned changes for v15 are also listed in the software changelog on our website.

I imagine if you turn the crank again (new products) it will get easier based on what you have learned.

Yep, the current user experience for the GigaWave sets the minimum bar for what we want to release in the future.

[joeqsmith]

- Implement SCPI standard. [Backwards-compatible with existing protocol.]

Are you planning to remove the older protocol once the new one has been adopted?   Will new products use the new protocol only?  Something different?   

The real advantage IMO to implementing SCPI is to use the current product as test case.   I doubt many people will write custom software for it and those that do will have already locked down their interface.     Down side is it adds complexity and increases the risk of there being problems. 

- Store jitter calibration in firmware, perform automatic jitter correction in software.

I though the current firmware stored the calibrations and that is why you have a unique release for each individual scope.   I can't imagine having to support something like this in production.

[SJL-Instruments]

New products will use SCPI only. As you said, the main benefits of SCPI for the GigaWave are (1) ease of use for new users, and (2) to be sure we have a bug-free implementation for future products. Whatever we implement, backwards compatibility is the #1 consideration and we will keep the current interface available.

Perhaps the current interface is already easy enough to get started with. We are the worst possible judge of whether this is true for new users, given how long we’ve spent thinking about the protocol. You did mention inconsistencies with the return terminators being a possible sticking point. SCPI would define all the low-level considerations like this in a standard way.

***
The current firmware does store the calibrations. There is no additional complexity needed to also store the jitter calibration. It’s not currently stored because it’s not necessary for the current functionality of the scope. From our perspective, this is a small change and straightforward to implement.

[joeqsmith]

Perhaps the current interface is already easy enough to get started with. We are the worst possible judge of whether this is true for new users, given how long we’ve spent thinking about the protocol. You did mention inconsistencies with the return terminators being a possible sticking point. SCPI would define all the low-level considerations like this in a standard way.

It will be interesting to see how you solve problems like stacking commands.  I had asked about it early on, then you took the stance of requiring a single command and waiting for an ack.

Concatenating commands
Multiple commands can be issued to an instrument in a single string. They are made of simple commands separated by a semicolon character (. For example, the command to "Measure a DC voltage then measure an AC current" would be issued as MEASure:VOLTage:DC?;:MEASure:CURRent:AC?.

Doing a quick search for the standard, came up with this one which is very old.
https://www.ivifoundation.org/downloads/SCPI/scpi-99.pdf