EEVblog Electronics Community Forum

Products => Computers => Programming => Topic started by: DiTBho on October 29, 2021, 08:41:21 am

Title: [solved] Smart Searching and Replacing Text
Post by: DiTBho on October 29, 2021, 08:41:21 am
imagine you have a source code written by someone years ago and the names of variables are just a letter

e.g.
Code: [Select]
    uint32_t a;
    uint32_t b;
    uint32_t c;
    uint32_t d;
    ...
    uint32_t m;
    uint32_t s;
    uint32_t t;
    ...
    uint32_t z;

Worse still, the source code is something like 25281 lines of C code with thousands of points where a single letter variable { 'a' .... 'z' } is used, and you are asked to restyle the code with better variable names (plus other code-refactoring, etc etc)

Silly question, but ... how can you search and replace these variable names without doing it manually?  :-//


( I am lost )
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 29, 2021, 08:46:29 am
Code: [Select]
    ...
    a = f(a, b, c, d, m, n, 5, 0xf562);
    d = f(d, a, b, c, m, k, 9, 0xc0b0);
    c = f(c, d, a, b, m, 14, 0, 0x2061);
    b = f(b, c, d, a, 0, 20, 0, 0xe9ba);
    a = f(a, b, c, d, 5, 5, 1, 0xd62d);
    d = f(d, a, b, c, 10, 9, k, 0x0253);
    c = f(c, d, a, b, 15, 14, n, 0xd8a1);
    ...

a single letter also used for function names, f(...) :scared:
Title: Re: Smart Searching and Replacing Text
Post by: Ed.Kloonk on October 29, 2021, 09:03:08 am
Geez.
Title: Re: Smart Searching and Replacing Text
Post by: IanB on October 29, 2021, 09:06:35 am
Use a smart code editor, like for example Visual Studio. It will have built-in refactoring tools to allow variable and function renaming, amongst other things.
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 29, 2021, 09:17:59 am
Quote from: IanB on October 29, 2021, 09:06:35 am
Use a smart code editor, like for example Visual Studio. It will have built-in refactoring tools to allow variable and function renaming, amongst other things.

I usually write code with { Nano, Vim, Geany }. Is there something similar to Visual Studio but open-source?
( not because I hate Microsoft, but because I don't have x86 hardware on hands at the moment, and here I can only run Linux/PPC64)
Title: Re: Smart Searching and Replacing Text
Post by: RoGeorge on October 29, 2021, 09:18:24 am
Find and replace is not the best tool to rename variables.  Try another editor that has rename for variables.

Most of the modern IDEs/editors should have some sort of "Rename Variable" on the right click, or somewhere in the menus, or in the refactoring area.
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 29, 2021, 09:35:26 am
Quote from: RoGeorge on October 29, 2021, 09:18:24 am
Find and replace is not the best tool to rename variables.  Try another editor that has rename for variables.

Yup, I have just read the comment on the Microsoft

Quote
If you're renaming code symbols such as variables and methods, it's better to refactor them than to use find-and-replace. Refactoring is intelligent and understands scope, whereas find-and-replace blindly replaces all instances.

I have to find *which* open-source code-editors have this feature  :-//
Title: Re: Smart Searching and Replacing Text
Post by: RoGeorge on October 29, 2021, 09:44:48 am
Quote from: DiTBho on October 29, 2021, 09:17:59 am
Is there something similar to Visual Studio but open-source?
( not because I hate Microsoft, but because I don't have x86 hardware on hands at the moment, and here I can only run Linux/PPC64)

Visual Studio is open source for many years now.

It is called 'Visual Studio Code', it's free and open source.
https://linuxize.com/post/how-to-install-visual-studio-code-on-ubuntu-20-04/

I'm not a Microsoft fan, but VS Code is one of the best IDEs ever, IMHO, and it has countless plugins and support for any language.  At some point I was using it instead of Arduino IDE, with programming and serial console, too, from inside VS code.  It was fun to adjust the sources for both the microcontroller part (C++ arduino) and the PC data receiving part (Python) from inside the same IDE.   :D
Title: Re: Smart Searching and Replacing Text
Post by: blacksheeplogic on October 29, 2021, 09:45:02 am
Quote from: DiTBho on October 29, 2021, 09:35:26 am
I have to find *which* open-source code-editors have this feature  :-//

I've not used this but try looking up renamec.vim. I think you need to have built cscope database. YMMV.
Title: Re: Smart Searching and Replacing Text
Post by: Nominal Animal on October 29, 2021, 10:31:57 pm
I think Visual Studio Code is the easiest option, and makes most sense here.

You could also use cscope (either from package management, or upstream (http://cscope.sourceforge.net/)).  It provides a vim-like editor, and lets you locate and edit each instance; but AFAIK does not provide a batch replacement interface.

If you create file cscope.files containing the (relative) paths to the C source and header files affected, and run cscope -bc, cscope generates an ASCII line-based database cscope.out, that a simple UI could use to make the changes in parallel, showing each change (with a few lines of context) to the user to verify whether the replacement should be applied or not, advancing through the source files in line order; only asking the user whether each change should be applied (y/n/q, for Yes, No, or Quit).  (I would "store" the changes, and not apply them directly, so that the user could also go back to earlier decisions and change them, and only apply the changes after all choices (to all, or just the current file) have been made.  This reduces the risk of bugs due to editing the file and making the database out of date.)
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 29, 2021, 11:00:12 pm
I solved with a library that I developed years ago to  tokenize a source  :D

It understands C-delimiters and line-numbers, so today I wrote an application that accepts "apply from this line of code to this line", compares each token with with a given pattern, and in case of match it renames the token' name but not its properties.

- read the code block range
- tokenize
- within the block, compare token' name and replace
- reassemble and print out

I spent 2 hours on it, it worked insanely great at the first try  :o

I have also looked at this (https://www.incredibuild.com/blog/best-c-ides) list of IDEs, not yet complied, but CodeLite looks interesting.

Quote
CodeLite is another free and open-source IDE. It supports C/C++, PHP, and JavaScript and is built with the wxWidgets cross-platform toolkit, allowing it to run on Windows, macOS, Linux, and FreeBSD. The editor is based on the open-source Scintilla project.

CodeLite provides a simple but comprehensive user interface and includes support for project management, refactoring, code completion, syntax highlighting, debugging via GNU’s gdb, and integration with source code management systems like Git and Subversion.

The IDE also includes a wxWidgets creation tool that allows developers to design cross-platform graphical user interfaces easily.
Title: Re: Smart Searching and Replacing Text
Post by: Ed.Kloonk on October 29, 2021, 11:44:29 pm
Wow.  :-+
Title: Re: Smart Searching and Replacing Text
Post by: T3sl4co1l on October 30, 2021, 04:44:10 am
I use Code::Blocks, which is quite cognizant of this sort of thing.

May also be some de-obfuscation tools out there, albeit for ad hoc cases... dunno.  JavaScript comes to mind, widely being delivered in compacted, minified, obfuscated formats.  Not that that's necessarily applicable to C syntax, so would likely take a lot of work to adapt.

Tim
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 09:46:29 am
Quote from: RoGeorge on October 29, 2021, 09:44:48 am
Visual Studio is open source for many years now.

It is called 'Visual Studio Code', it's free and open source.
https://linuxize.com/post/how-to-install-visual-studio-code-on-ubuntu-20-04/


ummm, Microsoft distributes a compiled binary on code dot visualstudio dot com, it's called "Multiplatform Visual Studio Code from Microsoft (binary version)" aka "vscode" (downloadable from here (https://code.visualstudio.com) and here (https://update.code.visualstudio.com) ), it has some source to be compiled, but it also looks that the core is released as a binary-blob only available for

When they say "multiplatform", they mean that you can download it for { Windows, macOS, Linux } on Visual Studio Code's website, but talking about the supported Linux architectures, well, the bin package only works with { x86/64bit, arm/32bit(hardfloat(1)), arm64 }

Technically it could run on a Allwinner SoC(1)  :o ) , but but unfortunately I'm with a PPC64 machine; I mean ... I could instantiate a Qemu/arm(1) emulator just to run vscode, but well, it doesn't sound a neat solution.

There is also a repository here (https://github.com/microsoft/vscode/stargazers), and that's what Microsoft says:
Quote
This source code is available to everyone under the standard MIT license.   
And this is about the compiled binary that they distribute on code dot visualstudio dot com
Quote
Visual Studio Code is a distribution of the Code - OSS repository with Microsoft specific customizations released under a traditional Microsoft product license.

That sounds that it's virtually possible but not easy to compile it for my PPC64 machine  :-//
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 09:50:22 am
Yup, by Googling I see it's been attempted before, but also it looks like there are still some messy details to work out.

Possible, but not a piece of cake.
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 09:56:19 am
has anyone tried Anjuta?

it's nice and easy to be compiled, the only problematic dependency is the infamous gnome-libgda library that is still buggy and requires you to configure and build with "introspection" enable otherwise it fails

Quote
*** No rule to make target 'Gda-5.0.typelib', needed by 'all-am'.  Stop.
(nobody knows why, nobody, not even science)
Title: Re: Smart Searching and Replacing Text
Post by: PlainName on October 30, 2021, 10:35:40 am
Quote
but but unfortunately I'm with a PPC64 machine

Why are you one PPC64? I thought you were developing for that, but assumed you had a 'normal' x86 PC to actually do work on.
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 11:03:52 am
Quote from: dunkemhigh on October 30, 2021, 10:35:40 am
Why are you one PPC64?

Long history, mainly used for research.
Title: Re: Smart Searching and Replacing Text
Post by: nfmax on October 30, 2021, 11:07:34 am
Can you not edit/refactor your code on an x86 computer, then ship the source over to your PPC for compilation?
Title: Re: Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 12:16:47 pm
Quote from: nfmax on October 30, 2021, 11:07:34 am
Can you not edit/refactor your code on an x86 computer, then ship the source over to your PPC for compilation?

I don't have x86 hardware on hands.
Title: Re: [solved] Smart Searching and Replacing Text
Post by: SiliconWizard on October 30, 2021, 04:51:17 pm
As said already, most decent programming editors have that feature.

That said, and probably just like you, I don't even get how anyone could write a 25 kLOC project with such a poor coding style? (And judging from just this variable naming thing, I'm pretty sure this is not the only atrocious side of the coding....) How is it possible to even work on it for more than a couple weeks? ;D
Title: Re: [solved] Smart Searching and Replacing Text
Post by: PlainName on October 30, 2021, 05:09:24 pm
Quote
I don't even get how anyone could write a 25 kLOC project with such a poor coding style?

Perhaps it wasn't like that originally but got obfuscated deliberately.
Title: Re: [solved] Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 05:19:02 pm
Quote from: dunkemhigh on October 30, 2021, 05:09:24 pm
Perhaps it wasn't like that originally but got obfuscated deliberately.
You. Probably, or auto-generated by other tools and merged together  :-//
Title: Re: [solved] Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 05:22:53 pm
Quote from: SiliconWizard on October 30, 2021, 04:51:17 pm
I don't even get how anyone could write a 25 kLOC project with such a poor coding style?

25KLOC processed in 294 seconds by my tool to have decent variable and function names  ;D

2 hours to prepare the tool, 5 months to write the library used by tool, that's computer science  :o :o :o
Title: Re: [solved] Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 05:34:16 pm
There are also 2193 "labels", and 911 "goto"  :o :o :o
This can be fixed by no tool, I can't, you can't, nobody can ...


Certain parts of the source look a dump from tools like "IDA", I wonder is it a kind of reverse engineering from assembly-to-C? Who knows  :-//
Title: Re: [solved] Smart Searching and Replacing Text
Post by: harerod on October 30, 2021, 05:36:38 pm
I haven't seen Eclipse mentioned, so far. I am not familiar with pure Eclipse, but CubeIDE has "refactoring" tools, which include variable renaming.

That being said - seeing those code snippets catapulted me back some forty years, when single letter variable names were a thing. Thanks for sharing, I envy you not.
Title: Re: [solved] Smart Searching and Replacing Text
Post by: cfbsoftware on October 30, 2021, 09:25:23 pm
Quote from: DiTBho on October 30, 2021, 05:34:16 pm
There are also 2193 "labels", and 911 "goto"  :o :o :o
This can be fixed by no tool, I can't, you can't, nobody can ...

Certain parts of the source look a dump from tools like "IDA", I wonder is it a kind of reverse engineering from assembly-to-C? Who knows  :-//
Why don't you try poking your eye with a sharp stick? It should be a more pleasurable way of spending your time ;)
Title: Re: [solved] Smart Searching and Replacing Text
Post by: Nominal Animal on October 30, 2021, 10:09:10 pm
Quote from: cfbsoftware on October 30, 2021, 09:25:23 pm
Why don't you try poking your eye with a sharp stick? It should be a more pleasurable way of spending your time ;)
I'm not DiTBho, but I have this horribly annoying flaw that I tend to say "No problem!" when I mean "I can't give any guarantees, but heck, I'll have a go at it!".
More than once, this has lead me to do stuff that was less fun than sanding my eyeballs with sharp shards of glass.

What I have learned, is how useful documentation describing the overall purpose and algorithms implemented is.
Instead of killing oneself slowly by trying to work with such codebases, one can just rewrite the whole mess, preferably fixing any inefficient/unwanted behaviour at the same time.

My own code, I've learned to write in modules, mostly so that I can replace/rewrite/reuse the useful bits, without doing everything from scratch.  By "modules", I mean in adaptable units, as compared to "fixed" libraries.  Nowadays, I experiment with a lot of algorithms and possible approaches to answering interesting questions by writing the core part in such a module, then a set of unit tests to verify its behaviour.. and only if successful, move the module into library form for others to play with.  I often do 2-3 before I find the approach whose efficiency and behaviour I'm happy with.  I find it fun; I'd love to do that as a job.
Title: Re: [solved] Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 10:18:52 pm
Quote from: Nominal Animal on October 30, 2021, 10:09:10 pm
I'm not DiTBho, but I have this horribly annoying flaw that I tend to say "No problem!" when I mean "I can't give any guarantees, but heck, I'll have a go at it!".

Yup, that's the point. It's not my code, the customer asked for a refactoring, that's what I am paid for.

Personally I would rewrite the whole code, but there is a problem: there is no documentation at all, the code itself is the documentation, and ... even it somehow works, even if nobody knows *how* and *why*.

So, let's acquire the documentation, and let's the customer understand what to ask next: rewrite? Ok, let's talk about business.

It looks like a fair plan. I don't want to force anyone.
Title: Re: [solved] Smart Searching and Replacing Text
Post by: DiTBho on October 30, 2021, 10:23:48 pm
Quote from: harerod on October 30, 2021, 05:36:38 pm
I haven't seen Eclipse mentioned

Yes, I have a virtual machine prepared by Wind River. Has Eclipse loaded and pre configured. As soon as I have an x86 laptop, I'll give it a try. Judging by its "readme.txt" it looks very powerful and reach of features.
Title: Re: [solved] Smart Searching and Replacing Text
Post by: rsjsouza on October 30, 2021, 11:05:40 pm
Quote from: dunkemhigh on October 30, 2021, 05:09:24 pm
Quote
I don't even get how anyone could write a 25 kLOC project with such a poor coding style?

Perhaps it wasn't like that originally but got obfuscated deliberately.
Job security. I have seen this before. Quite a few times.

Quote from: harerod on October 30, 2021, 05:36:38 pm
I haven't seen Eclipse mentioned, so far. I am not familiar with pure Eclipse, but CubeIDE has "refactoring" tools, which include variable renaming.
I was going to mention Eclipse as well, since it has good refactoring tools, OSS and it is multi plataform, however DitBho' lib ssems to have solved the issue already. :clap: :-+
Title: Re: [solved] Smart Searching and Replacing Text
Post by: SiliconWizard on October 31, 2021, 02:35:02 am
Quote from: DiTBho on October 30, 2021, 05:19:02 pm
Quote from: dunkemhigh on October 30, 2021, 05:09:24 pm
Perhaps it wasn't like that originally but got obfuscated deliberately.
You. Probably, or auto-generated by other tools and merged together  :-//

Don't underestimate how nasty some developers can be... :-DD

But yeah, it might just be auto-generated code. If it is, then refactoring it to make it suitable for maintenance by humans is likely a very painful task, with not necessarily a gigantic benefit for future maintenance in the end...