Author Topic: Is there any tool to convert hex files to int32?  (Read 2161 times)

0 Members and 1 Guest are viewing this topic.

Online ali_asadzadehTopic starter

  • Super Contributor
  • ***
  • Posts: 1896
  • Country: ca
Is there any tool to convert hex files to int32?
« on: March 28, 2019, 10:24:21 am »
Hi,
I have captured some data from ADC with my FPGA using UART, the output captured data is like this, I want to know if there is a readily available tool to convert the data which is in hex into int32 so that I can plot the data using matlab.

Note that the data is 24bit and I have added FF to separate the data, I can change the data format to 32bit by replacing the FF with 00, also the data is a signed data.
ASiDesigner, Stands for Application specific intelligent devices
I'm a Digital Expert from 8-bits to 64-bits
 

Offline Yansi

  • Super Contributor
  • ***
  • Posts: 3893
  • Country: 00
  • STM32, STM8, AVR, 8051
Re: Is there any tool to convert hex files to int32?
« Reply #1 on: March 28, 2019, 10:25:42 am »
That is not an HEX (intelHEX) file.

Just refactor the text using simple C program, written for purpose. Would get you somewhere within couple of tens of minutes. Parsing this is very easy. (even using directly matlab I'd say).

//EDIT: Also please note, that by "just adding FF or 00" in front of a 24bit two's complement (signed) number does not cut the mustard. You must properly sign extend.

FF FC B5 EE is correct, but FF 01 C4 6C is not, should be 00 01 C4 6C.  Copy the uppermost bit into the preceding byte.
« Last Edit: March 28, 2019, 10:30:45 am by Yansi »
 

Online ali_asadzadehTopic starter

  • Super Contributor
  • ***
  • Posts: 1896
  • Country: ca
Re: Is there any tool to convert hex files to int32?
« Reply #2 on: March 28, 2019, 10:43:49 am »
Quote
//EDIT: Also please note, that by "just adding FF or 00" in front of a 24bit two's complement (signed) number does not cut the mustard. You must properly sign extend.

FF FC B5 EE is correct, but FF 01 C4 6C is not, should be 00 01 C4 6C.  Copy the uppermost bit into the preceding byte.
Thanks for the hint, I'm aware of it ;)


Quote
Just refactor the text using simple C program, written for purpose. Would get you somewhere within couple of tens of minutes. Parsing this is very easy. (even using directly matlab I'd say).

I didn't want to write a c program for it, because I think it should be probably something out there that would do the job.
ASiDesigner, Stands for Application specific intelligent devices
I'm a Digital Expert from 8-bits to 64-bits
 

Offline Yansi

  • Super Contributor
  • ***
  • Posts: 3893
  • Country: 00
  • STM32, STM8, AVR, 8051
Re: Is there any tool to convert hex files to int32?
« Reply #3 on: March 28, 2019, 10:57:56 am »
Yes, there is, for example the matlab to begin with  ;)

https://se.mathworks.com/matlabcentral/answers/312599-how-do-i-parse-this-complex-text-file-with-textscan

It is as easy to do as in C. /* The C being likely the easier way*/
« Last Edit: March 28, 2019, 11:00:01 am by Yansi »
 
The following users thanked this post: ali_asadzadeh

Offline newbrain

  • Super Contributor
  • ***
  • Posts: 1714
  • Country: se
Re: Is there any tool to convert hex files to int32?
« Reply #4 on: March 28, 2019, 12:17:10 pm »
If you have a unix shell available (and chances are very high, among W10, Linux and macOS) you can use something as:
Code: [Select]
xxd -r -p dump.txt | od -t u4 -Anon the raw hex data (no header or footer: tail can come to the rescue).
This assumes the 24 bit data is big endian, and does not correct the sign and magnitude (I think that is easier in matlab)
man xxd
man od

Still, I would write a short C program...
Nandemo wa shiranai wa yo, shitteru koto dake.
 

Offline agehall

  • Frequent Contributor
  • **
  • Posts: 381
  • Country: se
Re: Is there any tool to convert hex files to int32?
« Reply #5 on: March 28, 2019, 01:04:02 pm »
I'm surprised so many suggest C. Isn't this exactly what your average scripting language is for? Python, PHP, Pike, JavaScript, Perl etc etc.

Anyhow, a quick custom piece of code is probably the way to go so that you have easy control over the parsing and the output.
 

Offline NorthGuy

  • Super Contributor
  • ***
  • Posts: 3137
  • Country: ca
Re: Is there any tool to convert hex files to int32?
« Reply #6 on: March 28, 2019, 01:14:32 pm »
How are you going to use the converted data? Read it into some sort of program? Then just modify your program to include parsing. It's 10 minutes of work.
 

Offline newbrain

  • Super Contributor
  • ***
  • Posts: 1714
  • Country: se
Re: Is there any tool to convert hex files to int32?
« Reply #7 on: March 28, 2019, 01:29:15 pm »
I'm surprised so many suggest C. Isn't this exactly what your average scripting language is for? Python, PHP, Pike, JavaScript, Perl etc etc.
You are probably right: I was thinking of doing it in PowerShell just for kicks...but then in a 10 minutes break I concocted this (use as a filter on command line):
Code: [Select]
#include <stdio.h>
#include <string.h>

#define SIZE 1024 /* Maximum allowed line length */
char lineBuffer[SIZE];

int main(void)
{
    /* Skip header - 3 lines */
    for (int i = 0; i < 3; i++)
        fgets(lineBuffer, SIZE, stdin);
    /* Stop at first short line (or EOF) */
    while(fgets(lineBuffer, SIZE, stdin) && (strlen(lineBuffer) > 2))
    {
        char *s = lineBuffer;
        unsigned short int l, m, h;
        /* Parse groups, ignore delimiter value, stop at first problem  */
        while(3 == sscanf(s, "%*X %hX %hX %hX", &h, &m, &l))
        {
            /* Assume big endian */
            int value = (h << 16) + (m << 8) + l;
            if(value >= 0x800000)
                value -= 0x1000000; /* Adjust negative numbers */
            printf("%d\n", value);  /* Write output */
            s += 12;                /* Next group */
        }
    }
    return 0;
}

It's throwaway code, not much error checking or style, HTH though...
Nandemo wa shiranai wa yo, shitteru koto dake.
 

Offline rstofer

  • Super Contributor
  • ***
  • Posts: 9886
  • Country: us
Re: Is there any tool to convert hex files to int32?
« Reply #8 on: March 28, 2019, 02:40:13 pm »
As mentioned above, there is a simple MATLAB program for doing this.  But MATLAB costs money so maybe it can be ported to Octave.

Octave has hex2dec() so you can do something like hex2dec(ff) and get back 255.  I didn't pursue it further...
 

Offline amyk

  • Super Contributor
  • ***
  • Posts: 8240
Re: Is there any tool to convert hex files to int32?
« Reply #9 on: March 29, 2019, 02:59:26 am »
I'm surprised so many suggest C. Isn't this exactly what your average scripting language is for? Python, PHP, Pike, JavaScript, Perl etc etc.

Anyhow, a quick custom piece of code is probably the way to go so that you have easy control over the parsing and the output.
If you're working with embedded stuff, chances are that you're familiar enough with C to do it easily.

Bit/byte manipulation is notably not very straightforward in scripting languages, because they were designed for higher-level tasks.

Here's my 3-minute attempt:
Code: [Select]
#include <stdio.h>
main() {
 int a, b, c, d;
 while(scanf("%x %x %x %x ", &a, &b, &c, &d) == 4) {
  a = b & 128 ? 255 : 0;
  a = (a << 24) | (b << 16) | (c << 8) | d;
  fwrite(&a, 1, 4, stdout);
 }
}
It would take me much longer to search for an existing program to do it.
 

Offline agehall

  • Frequent Contributor
  • **
  • Posts: 381
  • Country: se
Re: Is there any tool to convert hex files to int32?
« Reply #10 on: March 29, 2019, 06:09:44 am »
Bit/byte manipulation is notably not very straightforward in scripting languages, because they were designed for higher-level tasks.

You have obviously been using the wrong scripting languages then. :) Sane ones are pretty much the same as C for those operations.
 
The following users thanked this post: janoc

Offline mariush

  • Super Contributor
  • ***
  • Posts: 4983
  • Country: ro
  • .
Re: Is there any tool to convert hex files to int32?
« Reply #11 on: March 29, 2019, 08:03:13 am »
Here's how you would do it in PHP :

In Windows, just download php in a folder, for example C:\Programs\PHP and then open a command prompt in the same folder with your php script and text file and say
c:\programs\php\php.exe script.php

Change file name in the script if you want. Script just dumps each number on one line. You can say php.exe script.php >output.txt to redirect console output to a text file.
You can modify the script so that instead of a new line, the script prints a "," or  tab ("\t") or whatever character you want.

Code: [Select]
<?php

$ENTER 
"\x0D\x0A";  // Windows uses CRLF , Linux and only one character... change this as needed

// determine the path from where the script runs , and change all \ slashes to / slashes - Windows can work with either.

$folderRoot str_replace('\\','/',dirname(__FILE__));
if (
substr($folderRoot,strlen($folderRoot)-1,1)!=='/'$folderRoot .='/';

// change the file name here or replace with $argv[1] to read the file name from command line ex php.exe script.php filename.txt  -> $argv[1]=filename.txt 
// though it will only work if files have no spaces in their names

$content file_get_contents($folderRoot.'20190328142916.txt');

$lines explode($ENTER,$content);
$v = array();
$vindex 0;
foreach (
$lines as $line) {
if (isHexLine($line)==TRUE) {
$values explode(' ',$line);
foreach ($values as $value) {
if ((ctype_xdigit($value)==TRUE) && (strlen($value)==2)) { // hexadecimal , one byte
$nr hexdec($value);
$v[$vindex] = $nr;
$vindex++;
if ($vindex==4) { // read 4 bytes 
// comment next line if you don't want the first byte to be reset
$v[0] = 0;
$nr = ($v[0]<<24) + ($v[1]<<16) +($v[2]<<8) + $v[3];
echo $nr.$ENTER;
$v[0] = 0$v[1]=0;$v[2]=0;$v[3]=0;
$vindex=0;
}
}
}
}
}



// function checks if the line of text is composed of only hexadecimal characters or space
function isHexLine($text) {
if (strlen($text)==0) return FALSE;
for ($i=0;$i<strlen($text);$i++) {
$c substr($text,$i,1);
if (($c==' ') || (ctype_xdigit($c)==TRUE)) {
// it's space or hexadecimal char, so continue
} else {
return FALSE// bad char, don't process this line
}
}
return TRUE;
}
?>

« Last Edit: March 29, 2019, 08:09:38 am by mariush »
 

Online ali_asadzadehTopic starter

  • Super Contributor
  • ***
  • Posts: 1896
  • Country: ca
Re: Is there any tool to convert hex files to int32?
« Reply #12 on: March 29, 2019, 08:13:47 am »
Wow...really thanks guys, I did not expect that, really appreciate the tips. though I'm a fan of Node and Javascript
ASiDesigner, Stands for Application specific intelligent devices
I'm a Digital Expert from 8-bits to 64-bits
 

Online ali_asadzadehTopic starter

  • Super Contributor
  • ***
  • Posts: 1896
  • Country: ca
Re: Is there any tool to convert hex files to int32?
« Reply #13 on: March 29, 2019, 11:02:16 am »
So for completeness I have written the Node version too

Code: [Select]
<?php
const fs = require('fs');
const 
EOL = require('os').EOL;

const 
contents fs.readFileSync('./20190328142916.txt');
const 
reg = /FF\s+(\w{2})\s+(\w{2})\s+(\w{2})/g;

let group reg.exec(contents);
let combinedDecimalValues '';

while (
group) {
    const 
hexValue = `${group[1]}${group[2]}${group[3]}`;
    const 
parsedValue parseInt(hexValue16);
    
let decimalValue parsedValue;

    if (
parsedValue 0x7FFFFF) {
        
decimalValue parsedValue 0xFFFFFF 1;
    }

    
combinedDecimalValues += decimalValue EOL;

    
group reg.exec(contents);
}

fs.writeFileSync('./output.txt'combinedDecimalValues);
mariush how did you highlighted your code?
Do we have code highlight in here?
« Last Edit: March 29, 2019, 03:16:33 pm by ali_asadzadeh »
ASiDesigner, Stands for Application specific intelligent devices
I'm a Digital Expert from 8-bits to 64-bits
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 6733
  • Country: pl
Re: Is there any tool to convert hex files to int32?
« Reply #14 on: March 29, 2019, 02:29:42 pm »
That's how:
Code: [Select]
<?php
const fs = require('fs');
const 
EOL = require('os').EOL;

const 
contents fs.readFileSync('./20190328142916.txt');
const 
reg = /FF\s+(\w{2})\s+(\w{2})\s+(\w{2})/g;

let group reg.exec(contents);
let combinedDecimalValues '';

while (
group) {
    const 
hexValue = `${group[1]}${group[2]}${group[3]}`;
    const 
parsedValue parseInt(hexValue16);
    
let decimalValue parsedValue;

    if (
parsedValue 0x7FFFFF) {
        
decimalValue parsedValue 0xFFFFFF 1;
    }

    
combinedDecimalValues += decimalValue EOL;

    
group reg.exec(contents);
}

fs.writeFileSync('./output.txt'combinedDecimalValues);
:-DD

In general, you can click Quote to obtain "source code" of other people's posts.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 8110
  • Country: fi
Re: Is there any tool to convert hex files to int32?
« Reply #15 on: March 29, 2019, 03:40:34 pm »
I'm surprised so many suggest C. Isn't this exactly what your average scripting language is for? Python, PHP, Pike, JavaScript, Perl etc etc.

Nothing surprising here. It's a 5-minute throwaway job on any half-decent language. It's approx 10-20 lines on any language, Python or C.

The result (how easy and fast it is) depends entirely on the language skills and the current mindset of the programmer. If they have written C for an embedded project in last years, trying to do it in Python because it's somehow "easier" could end up being a disaster in time usage.

Really almost any language will do. Trying to find some academical "optimally easy" solution doesn't make a lot of sense if it isn't easy in your specific case.

When you have a round peg and a round hole and need to apply just a little bit of extra force to get it in, it doesn't matter much which kind of hammer you end up using, most will do, and you'll use what you have already in your hand, as long as it isn't clearly wrong. Shopping for an optimum type of hammer would make zero sense.

For a beginner, Python - as well as many other scripting languages - are good choices, if they don't know any language yet, and need to start by learning the language and the tools.
« Last Edit: March 29, 2019, 03:44:47 pm by Siwastaja »
 
The following users thanked this post: newbrain


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf