Program that can log/control many multimeters and other devices.

[MrPete]

It appears also, if I don't want to modify values but DO want to incorporate additional columns of data, my choices are either:
a) Use expressions that work inside Math or a live script during data collection (see https://lygte-info.dk/project/TestControllerMath%20UK.html#Including_a_control_variable_in_the_data )
b) After data collection is complete, my only option is to fully replace the entire table (eg tableAddCSVText()) or redo the table from scratch one row at a time (tableInitHeader() and tableAddRow() ).

Does that sound correct?

You can add math expression after the logging is done, math is only blocked while logging is running.

Great for others. Thanks for confirming this!

For my use case, even a custom math expression can't process multiple rows (eg calculate a trend).

So, I've rewritten my applet to be a standalone script. I've hit the following exception when writing a new table. I set the third parameter to 1, to recreate the index column. Not sure why this produces Array Index Out Of Bounds? Do I need to incorporate an "Index" header?

#calc
java.lang.ArrayIndexOutOfBoundsException: 12
        at dk.hkj.database.DataRow.load(DataRow.java:144)
        at dk.hkj.database.DataBase.loadFromList(DataBase.java:349)
        at dk.hkj.database.ScriptFunctions$30.execute(ScriptFunctions.java:988)
        at dk.hkj.script.Script.parseValue(Script.java:279)
        at dk.hkj.script.Script.parseBitOp(Script.java:326)
        at dk.hkj.script.Script.parseProduct(Script.java:398)
        at dk.hkj.script.Script.parseSum(Script.java:480)
        at dk.hkj.script.Script.parseCompare(Script.java:535)
        at dk.hkj.script.Script.parseLogical(Script.java:571)
        at dk.hkj.script.Script.parseQuestionMark(Script.java:597)
        at dk.hkj.script.Script.parseExpression(Script.java:622)
        at dk.hkj.script.Script.statement(Script.java:1109)
        at dk.hkj.script.Script.doExecute(Script.java:1260)
        at dk.hkj.script.Script.execute(Script.java:1282)
        at dk.hkj.script.Script.execute(Script.java:1288)
        at dk.hkj.main.CommandProcessor.sysCmdCalc(CommandProcessor.java:2426)
        at dk.hkj.main.CommandProcessor.processCommands(CommandProcessor.java:2986)
        at dk.hkj.main.CommandProcessor$CmdEnvironment.run(CommandProcessor.java:369)
        at dk.hkj.main.CommandProcessor$BackgroundCommands.run(CommandProcessor.java:1789)
;; #calc failed due to 12


// 6. Inject the data matrix
tableAddCSVText(masterTsvBuffer, "", 1);____<----____

[MrPete]

@HKJ - another minor bug. And a less minor challenge...

MINOR BUG

I've loaded Table with some saved data.
If I then UNcheck the "dateTime" selector on the right side of Table...
   - The dateTime column DOES disappear
   - BUT the column header mouseover popup still reflects the original column sequence.
Result: every column from where dateTime to the end now has incorrect column header mouseover text.

NOT SO MINOR
The following data chain causes dateTime data to become mangled:

a) Record data in the normal way from live devices. Shows up in Table correctly
b) Export the data to a CSV file. The CSV version of dateTime is a Unix timestamp in seconds (and fractions) since 1/1/1970
c) Load the data back into Table. It is correctly restored
d) Pull the data programmatically via table(<row>,<col#>) - in this case, the data is formatted as yyyymmddhhmmss
    (assemble the data with intervening tabs into a tab-delimited CSV buffer)
e) Reload table from the CSV buffer: tableAddCSVText(masterTsvBuffer, "", 0);

RESULT: the dateTime column now has values like 31/01-644003 06:32:02

(My guess: the import still wants Unix timestamps, not yyyymmddhhmmss like what table(<row>,<col>) produces)

Do you consider this a bug, or something that just needs documentation and some kind of timestamp conversion during scripted export/import?

[HKJ]

For my use case, even a custom math expression can't process multiple rows (eg calculate a trend).

So, I've rewritten my applet to be a standalone script. I've hit the following exception when writing a new table. I set the third parameter to 1, to recreate the index column. Not sure why this produces Array Index Out Of Bounds? Do I need to incorporate an "Index" header?

#calc
java.lang.ArrayIndexOutOfBoundsException: 12
        at dk.hkj.database.DataRow.load(DataRow.java:144)
        at dk.hkj.database.DataBase.loadFromList(DataBase.java:349)

The table and all arrays are 0 based, i.e. if they have 3 columns, they are number 0, 1, 2 and 3 is Index out of bounds.

[HKJ]

@HKJ - another minor bug. And a less minor challenge...

MINOR BUG

I've loaded Table with some saved data.
If I then UNcheck the "dateTime" selector on the right side of Table...
   - The dateTime column DOES disappear
   - BUT the column header mouseover popup still reflects the original column sequence.
Result: every column from where dateTime to the end now has incorrect column header mouseover text.

I have fixed it, it will be included in next update, test or release.
It did also affect search in table and format changes.

NOT SO MINOR
The following data chain causes dateTime data to become mangled:
d) Pull the data programmatically via table(<row>,<col#>) - in this case, the data is formatted as yyyymmddhhmmss
    (assemble the data with intervening tabs into a tab-delimited CSV buffer)
e) Reload table from the CSV buffer: tableAddCSVText(masterTsvBuffer, "", 0);

RESULT: the dateTime column now has values like 31/01-644003 06:32:02

(My guess: the import still wants Unix timestamps, not yyyymmddhhmmss like what table(<row>,<col>) produces)

Do you consider this a bug, or something that just needs documentation and some kind of timestamp conversion during scripted export/import?

It is not really a bug, but is related to how variable works, especial a dateTime data type. It has the unix value and a format specification that is used when you request the values as a string. Try typecast it to double(), then I expect you will get the number.

[MrPete]

For my use case, even a custom math expression can't process multiple rows (eg calculate a trend).

So, I've rewritten my applet to be a standalone script. I've hit the following exception when writing a new table. I set the third parameter to 1, to recreate the index column. Not sure why this produces Array Index Out Of Bounds? Do I need to incorporate an "Index" header?

#calc
java.lang.ArrayIndexOutOfBoundsException: 12
        at dk.hkj.database.DataRow.load(DataRow.java:144)
        at dk.hkj.database.DataBase.loadFromList(DataBase.java:349)

The table and all arrays are 0 based, i.e. if they have 3 columns, they are number 0, 1, 2 and 3 is Index out of bounds.

I get that. What I am saying is, the function tableAddCSVText(csv,columnNames,addIndex) is documented with "if the last parameter is non-zero a index column is added before the data."

What I described above is: if I set the third addIndex parameter to 1, then I get the above Index out of bounds exception. If I set it to zero, tableAddCSVText works, but no index column is created of course.

[MrPete]

@HKJ - another minor bug. And a less minor challenge...

MINOR BUG

I've loaded Table with some saved data.
If I then UNcheck the "dateTime" selector on the right side of Table...
   - The dateTime column DOES disappear
   - BUT the column header mouseover popup still reflects the original column sequence.
Result: every column from where dateTime to the end now has incorrect column header mouseover text.

I have fixed it, it will be included in next update, test or release.
It did also affect search in table and format changes.

NOT SO MINOR
The following data chain causes dateTime data to become mangled:
d) Pull the data programmatically via table(<row>,<col#>) - in this case, the data is formatted as yyyymmddhhmmss
    (assemble the data with intervening tabs into a tab-delimited CSV buffer)
e) Reload table from the CSV buffer: tableAddCSVText(masterTsvBuffer, "", 0);

RESULT: the dateTime column now has values like 31/01-644003 06:32:02

(My guess: the import still wants Unix timestamps, not yyyymmddhhmmss like what table(<row>,<col>) produces)

Do you consider this a bug, or something that just needs documentation and some kind of timestamp conversion during scripted export/import?

It is not really a bug, but is related to how variable works, especial a dateTime data type. It has the unix value and a format specification that is used when you request the values as a string. Try typecast it to double(), then I expect you will get the number.

Here's the real answer:

Code: [Select]

=table(5,"dateTime")
;; 20260607203921
=double(table(5,"dateTime"))
;; 20612.11066127315
=long(table(5,"dateTime"))
;; 1780886361134Typecast to long appears to be what the CSV importer is expecting. Nice: that is a value in Unix milliseconds, and DOES incluide the fractional seconds portion We should document that for future users.

(I would also suggest that perhaps the table() reader should offer a mode or option that directly provides data that can be re-injected across the board w/o typecasting. This will avoid a lot of niggly code on the user side. The system knows all about each column. Being able to programmatically save/load table data without undue pain seems a valuable goal.

Which leads to an obvious (?) question:
)
When performing programmatic loading of the table, how should extracted tableColumnUnit() data be injected into the new table?

[HKJ]

I get that. What I am saying is, the function tableAddCSVText(csv,columnNames,addIndex) is documented with "if the last parameter is non-zero a index column is added before the data."

What I described above is: if I set the third addIndex parameter to 1, then I get the above Index out of bounds exception. If I set it to zero, tableAddCSVText works, but no index column is created of course.

Ok, the name of the index column is supposed to be the first item in columnName, but I have changed the code so a empty columnName will add "index".

[HKJ]

Here's the real answer:

Code: [Select]

=table(5,"dateTime")
;; 20260607203921
=double(table(5,"dateTime"))
;; 20612.11066127315
=long(table(5,"dateTime"))
;; 1780886361134Typecast to long appears to be what the CSV importer is expecting. Nice: that is a value in Unix milliseconds, and DOES incluide the fractional seconds portion We should document that for future users.

(I would also suggest that perhaps the table() reader should offer a mode or option that directly provides data that can be re-injected across the board w/o typecasting. This will avoid a lot of niggly code on the user side. The system knows all about each column. Being able to programmatically save/load table data without undue pain seems a valuable goal.

Documentation adjusted.

Which leads to an obvious (?) question:
)
When performing programmatic loading of the table, how should extracted tableColumnUnit() data be injected into the new table?

It is not really supported at the current time, maybe it would be a good idea to add it.
Uusally TC will check the device database for handles that matches the first part of the column name.

[MrPete]

I get that. What I am saying is, the function tableAddCSVText(csv,columnNames,addIndex) is documented with "if the last parameter is non-zero a index column is added before the data."

What I described above is: if I set the third addIndex parameter to 1, then I get the above Index out of bounds exception. If I set it to zero, tableAddCSVText works, but no index column is created of course.

Ok, the name of the index column is supposed to be the first item in columnName, but I have changed the code so a empty columnName will add "index".

Are you saying I should leave an empty first column to be filled by TC? That's not what I would have guessed. My assumption was, TC would insert an additional column before my columns, containing an index.  That's how I read "index column is added before the data."

[HKJ]

I get that. What I am saying is, the function tableAddCSVText(csv,columnNames,addIndex) is documented with "if the last parameter is non-zero a index column is added before the data."

What I described above is: if I set the third addIndex parameter to 1, then I get the above Index out of bounds exception. If I set it to zero, tableAddCSVText works, but no index column is created of course.

Ok, the name of the index column is supposed to be the first item in columnName, but I have changed the code so a empty columnName will add "index".

Are you saying I should leave an empty first column to be filled by TC? That's not what I would have guessed. My assumption was, TC would insert an additional column before my columns, containing an index.  That's how I read "index column is added before the data."

Sort of, when you specify the columns:
Data: value1, value2, value3
You specification when doing automatic index addition: index, value1, value2, value3

The behavior when reading the header line from the file was not useable, that is the reason I changed it.

[MrPete]

Sort of, when you specify the columns:
Data: value1, value2, value3
You specification when doing automatic index addition: index, value1, value2, value3

The behavior when reading the header line from the file was not useable, that is the reason I changed it.

To be precise, is this the new way you've coded it?

Assume I have a CSV memory buffer with N columns of CSV data.

If Auto-Index is enabled, any of these will work:
  a) call with same data in 1st parameter, and columnNames param with "index," pre-pended
OR
  b) call with same data in 1st parameter, and empty columnName param. TC will note that there's no "index" column, and will insert it.
OR
  c) insert a blank column before data columns in 1st parameter, with header name "index". Empty columname param. TC will see "index" and fill that column.

[MrPete]

@HKJ I am encountering additional CSV import challenges.

To solve this, I want to write my data buffer to a file. I see ways to LOG to a file, but don't see a simple file write function in the programming language.

Is there a way to do this? I have a BigStringVariable and want to write its content to a file.

Never mind - found it. fileAppendText() and fileAppendClose();

[HKJ]

To be precise, is this the new way you've coded it?

Assume I have a CSV memory buffer with N columns of CSV data.

If Auto-Index is enabled, any of these will work:
  a) call with same data in 1st parameter, and columnNames param with "index," pre-pended
OR
  b) call with same data in 1st parameter, and empty columnName param. TC will note that there's no "index" column, and will insert it.
OR
  c) insert a blank column before data columns in 1st parameter, with header name "index". Empty columname param. TC will see "index" and fill that column.

The two formats supported is:
"index";"time";"minutes";"dateTime";"VSG.Sine"
0;0,016;0,00026666666667;1781377870,893;0
1;0,026;0,00043333333333;1781377870,902;0,06283
2;0,038;0,00063333333333;1781377870,914;0,13823
3;0,046;0,00076666666667;1781377870,922;0,18848
or
0;0,016;0,00026666666667;1781377870,893;0
1;0,026;0,00043333333333;1781377870,902;0,06283
2;0,038;0,00063333333333;1781377870,914;0,13823
3;0,046;0,00076666666667;1781377870,922;0,18848
With a columnNames param: "\"index\";\"time\";\"minutes\";\"dateTime\";\"VSG.Sine\""
Note the \" is required because the parser will use a single "

If you want to add index to the two above:
With the current version it is not possible with the first example, but the updated version will add a "index" column at the start, i.e. there will be two index columns.
For the second example you must modify the header line to: "\"myIndex\",\"index\";\"time\";\"minutes\";\"dateTime\";\"VSG.Sine\""

[HKJ]

@HKJ I am encountering additional CSV import challenges.

To solve this, I want to write my data buffer to a file. I see ways to LOG to a file, but don't see a simple file write function in the programming language.

Is there a way to do this? I have a BigStringVariable and want to write its content to a file.

Never mind - found it. fileAppendText() and fileAppendClose();

The tableAddCSVText() uses the same code as loading a file, but gives you the flexibility to add a header in code.

[MrPete]

The two formats supported is:
"index";"time";"minutes";"dateTime";"VSG.Sine"
0;0,016;0,00026666666667;1781377870,893;0
1;0,026;0,00043333333333;1781377870,902;0,06283
2;0,038;0,00063333333333;1781377870,914;0,13823
3;0,046;0,00076666666667;1781377870,922;0,18848
or
0;0,016;0,00026666666667;1781377870,893;0
1;0,026;0,00043333333333;1781377870,902;0,06283
2;0,038;0,00063333333333;1781377870,914;0,13823
3;0,046;0,00076666666667;1781377870,922;0,18848
With a columnNames param: "\"index\";\"time\";\"minutes\";\"dateTime\";\"VSG.Sine\""
Note the \" is required because the parser will use a single "

If you want to add index to the two above:
With the current version it is not possible with the first example, but the updated version will add a "index" column at the start, i.e. there will be two index columns.
For the second example you must modify the header line to: "\"myIndex\",\"index\";\"time\";\"minutes\";\"dateTime\";\"VSG.Sine\""

A few notes and updates on this, based on testing:
1) I'm not sure what format you're describing above. My guess: your example is semicolon-separated (";") -- perhaps because that is needed in parts of the world where fractional numbers are "123,45" instead of "123.45"

2) I've got examples with no quotes around the column header names. Of course they ARE needed if certain characters are embedded. One of the simplest formats is tab-separated. That's compatible with all number formats, and requires no quoting.

MOST important: What we determined above about dateTime conversion is wrong. It is NOT enough to typecast table(n,"dateTime") to long!

What the importer expects is seconds with fractions. While some unixes auto-recognize both formats, right now the TC importer does not

If I convert the data as follows, it works, both using the CSV table loader and if I load into table from a file:  ...long(table(n, "dateTime"))/1000.0;

[HKJ]

A few notes and updates on this, based on testing:
1) I'm not sure what format you're describing above. My guess: your example is semicolon-separated (";") -- perhaps because that is needed in parts of the world where fractional numbers are "123,45" instead of "123.45"

That is the csv format that is used around here, but as you already know TC support both common csv formats.

2) I've got examples with no quotes around the column header names. Of course they ARE needed if certain characters are embedded. One of the simplest formats is tab-separated. That's compatible with all number formats, and requires no quoting.

My example was just copied from a csv file.

MOST important: What we determined above about dateTime conversion is wrong. It is NOT enough to typecast table(n,"dateTime") to long!

What the importer expects is seconds with fractions. While some unixes auto-recognize both formats, right now the TC importer does not

If I convert the data as follows, it works, both using the CSV table loader and if I load into table from a file:  ...long(table(n, "dateTime"))/1000.0;

The importer can also handle windows data-time format, if the field is not fully numeric it will try the current windows data-time format.

[MrPete]

MOST important: What we determined above about dateTime conversion is wrong. It is NOT enough to typecast table(n,"dateTime") to long!

What the importer expects is seconds with fractions. While some unixes auto-recognize both formats, right now the TC importer does not

If I convert the data as follows, it works, both using the CSV table loader and if I load into table from a file:  ...long(table(n, "dateTime"))/1000.0;

The importer can also handle windows data-time format, if the field is not fully numeric it will try the current windows data-time format.

Unfortunately, table() doesn't produce that either. At least typecast to long and divide by 1000 is a data conversion path that works

[HKJ]

Unfortunately, table() doesn't produce that either. At least typecast to long and divide by 1000 is a data conversion path that works

I will consider adding a function to do it in one operation.

[MrPete]

@HJK, is there a programming limitation on nested while loops?

I have the following code. It dies (no exception - just 'stuck' waiting for me to click the Abort button) on the inner break statement.

The logical nesting is:

while
   if
      while
           if
               break
           endif;
       endwhile;
    // I expect the break to continue here. But nothing happens...
...
(not shown: later endif and endwhile

Code: [Select]

loopIdx = 1;
while loopIdx < (totalSamples-10) do
return0(printLog("loop "+loopIdx));
if test1 && test2
var loopIdxR = loopIdx;
while loopIdxR < (totalSamples-10) do
return0(printLog("loopR "+loopIdxR));
if abs(cDSlopeArray[loopIdxR]) < 0.0001
return0(printLog("found stable, about to break at "+loopIdxR));
break;
endif;

loopIdxR = loopIdxR + 1;

endwhile;

return0(printLog("found stable at "+loopIdxR));

[HKJ]

@HJK, is there a programming limitation on nested while loops?

Not to my knowledge.
What you found was a bug and I hope it is fixed now.
You can download a new test version.

[flash2b]

I have a GPIB device, not SCPI compliant so Ascii driver.

I need to send [CLR] to the device and wait for a bit and then I need to send a command to it to get the device id.

I use #initCmd

Will this be executed before model? and what do I need to do?

#initCmd [CLR];[5000] does not show any clear statement in the debug log and also the meter does not init.

[HKJ]

I have a GPIB device, not SCPI compliant so Ascii driver.

I need to send [CLR] to the device and wait for a bit and then I need to send a command to it to get the device id.

I use #initCmd

Will this be executed before model? and what do I need to do?

#initCmd [CLR];[5000] does not show any clear statement in the debug log and also the meter does not init.

#initCmd is execute right after *idn?
There is no way to get around that.

[flash2b]

Yes I found that out, I need to send CLR before identification. So there is no way around that?

also is it [CLR] or tx [CLR] ?

I can sent ++clr from the AR488, but I need to wait a while. can I delay #verifydevice ?

[HKJ]

Yes I found that out, I need to send CLR before identification. So there is no way around that?

also is it [CLR] or tx [CLR] ?

I can sent ++clr from the AR488, but I need to wait a while. can I delay #verifydevice ?

In SCPIx either way works.

[flash2b]

But I use ascii.. so tx [CLR] ?

Is there a way to delay the #verifydevice after the AR488 does its initialisation ?