-
Bash is good, but it's too error prone.
What would you use to replace it?
(excluding big elephants like Python, which I like, but it consumes too many resources)
Php ... well, it looked a good idea, after 1 year, let me say: never again!
Lua? eLua?
...
Looking for something I can use to replace my startup system scripts (/etc/init.d/*), as well as my system scripts (in /usr/bin/), currently all written in Bash-scripting.
-
What is wrong with bash?
Perl was explicitly designed to be used instead of shell scripting. It's more or a real programming language, but with convenience features to let you use it almost like shell.
Ruby is also very good at replacing shell scripting, especially once you "require 'fileutils'", and has a lot more of the shell/perl spirit than Python does. And is just simply a better language than Python.
TCL suffers from the same problems arising from being based on text replacement as shell.
-
I know eLua (e="e"embedded) has been used in SONY Playstation video games, embedded into the game engine.
Some text editors come with eLua built-in. Used to provide advanced (and programmable) macros facilities.
I am tempted to open a branch, replacing all the boot-scripts with Lua scripts.
You say tcl ... umm interesting too ;D
-
What is wrong with bash?
functions are not actually functions
-> can override without a warning
-> no actual return value
-> the number of arguments is not checked
-> arguments data-type is not checked
the integer data type is simulated by external tools
-> compare two numbers is ugly, and too error prone
-
Ruby is also very good at replacing shell scripting, especially once you "require 'fileutils'", and has a lot more of the shell/perl spirit than Python does. And is just simply a better language than Python.
seems *very* interesting.
I have to understand
- how fast it is
- how much ram it consumes
First, let learn its basic usage :D
-
seems *very* interesting.
I have to understand
- how fast it is
- how much ram it consumes
First, let learn its basic usage :D
Be careful, it's way too easy to fall in love with Ruby.
-
functions are not actually functions
-> can override without a warning
-> no actual return value
-> the number of arguments is not checked
-> arguments data-type is not checked
the integer data type is simulated by external tools
-> compare two numbers is ugly, and too error prone
Those may be arguments not to want to write shell scripts, but not to rewrite scripts that will probably have been tested for years or even decades on many systems.
I believe most modern systems don't actually use bash for init scripts, but something lighter weight like ash. Sure, the language is ugly, but there's a reason why it has been used for simple scripts for decades.
Perl and Ruby would be the obvious candidates in my mind, but I'm not so sure if they are much lighter than Python. If you want very low footprint, try Forth. It's used in some bootloaders.
-
I'd be struggling to find a good reason to use anything but lightweight shell for boot scripts.
And that's usually not even bash -- for example, Debian has dash as its generic /bin/sh interpreter:
Description: POSIX-compliant shell
The Debian Almquist Shell (dash) is a POSIX-compliant shell derived
from ash.
.
Since it executes scripts faster than bash, and has fewer library
dependencies (making it more robust against software or hardware
failures), it is used as the default system shell on Debian systems.
It's also a good practice to avoid using bash-specific functionality in shell scripts. It makes them less portable.
-
In defense of bash on a couple of things...
...
-> no actual return value
There is a "return [n]" command which sets the exit status of a function. It can be tested by the caller in-line as 0 or non-0 (just like an external program call), or you can test for a specific return value by using $? in a subsequent conditional.
-> the number of arguments is not checked
If you need to check it, $# is local to the function and returns the argument count.
the integer data type is simulated by external tools
-> compare two numbers is ugly, and too error prone
Not sure what you mean by external tools, but comparing numbers using (( ... )) is built-in and follows C arithmetic evaluation syntax. It's pretty clean looking, IMO:
x=5
y=6
(( x < y )) && echo hi
# or if you prefer the longer form...
if (( x < y )); then
echo hi
fi
However, to add to your other list items, it really bugs me that bash doesn't support floats after all these years.
-
-> the number of arguments is not checked
If you need to check it, $# is local to the function and returns the argument count.
function foo() how many arguments does it need?!? undefined
{
local arg1="$1"
local arg2="$2"
local arg3="$3"
local argn=$#
...
}
then somewhere ... even far from the define (e.g. "source /opt/scripts/your_script"), you invoke foo "1" "2" what is arg3?!? empty, and do you always check argn?!? also, what if you "swap" two args?!?
I partially fixed this problem with "check_var", which invokes "panic" if a var is empty
function foo() how many arguments does it need?!? undefined
{
local fid="foo" <------------- isn't there a feature that automatically reports the function name?
local arg1="$1"
local arg2="$2"
local arg3="$3"
...
check_var "arg1" "$arg1" "$fid"
check_var "arg2" "$arg2" "$fid"
check_var "arg3" "$arg3" "$fid"
...
}
foo "1" "2"
[!] panic
in function foo var arg3 is empty
-
Not sure what you mean by external tools, but comparing numbers using (( ... )) is built-in and follows C arithmetic evaluation syntax.
However, to add to your other list items, it really bugs me that bash doesn't support floats after all these years.
I wrote a patch manager to facilitate kernel hacking.
I needed to comparing { binutils, gcc } version withing a { min ... max } range
and ended writing these bash functions
module="profile/do/utils_ver"
# 2.6.15 -> 002006015
# ver -> num
function ver_num_get()
{
local fid="ver_num_get"
local item="$1"
local cmp1
local cmp2
check_var "item" "$item" "$module" "$fid"
cmp1=$(echo "$item < 10" | bc -l)
cmp2=$(echo "$item < 100" | bc -l)
if [ "$cmp1" == "1" ]
then
ans="00$item"
else
if [ "$cmp2" == "1" ]
then
ans="0$item"
else
ans="$item"
fi
fi
# return ans
}
# ver x.y.z -> %03x%03y%03z
function ver_get()
{
local fid="ver_get"
local item="$1"
local is_done
local curr
local ifam
local isub
local iver
check_var "item" "$item" "$module" "$fid"
iver=""
curr="$item"
is_done="false"
while [ $is_done == "false" ]
do
ifam="`myfilename_deextendL $curr`"
isub="`myfilename_getextL $curr`"
ver_num_get $ifam
iver="$iver$ans"
curr="$isub"
if [ "*$curr" == "*" ]
then
is_done="true"
fi
done
ans="$iver"
# return ans
}
function ver_is_ok()
{
local fid="ver_is_ok"
local ver_cur="$1"
local ver_min="$2"
local ver_max="$3"
local cmp1
local cmp2
check_var "ver_cur" "$ver_cur" "$module" "$fid"
check_var "ver_min" "$ver_min" "$module" "$fid"
check_var "ver_max" "$ver_max" "$module" "$fid"
# convert into xxx format
ver_get "$ver_cur"
ver_cur="$ans"
ver_get "$ver_min"
ver_min="$ans"
ver_get "$ver_max"
ver_max="$ans"
# check check the range borders
cmp1=$(echo "$ver_cur >= $ver_min" | bc -l)
cmp2=$(echo "$ver_cur <= $ver_max" | bc -l)
if [ "*$cmp1" != "*1" ] || [ "*$cmp2" != "*1" ]
then
ans="false"
else
ans="true"
fi
#return ans
}
external tools required:
{
portage@sys-devel/bc:bc,
overlay@app-utils/filename_utils:{myfilename_deextendL, myfilename_getextL}
}
I wrote the overlay@app-utils/filename_utils package because I missed something similar in bash scripting.
(yes, the coding style is like ~my-c
I am too "my-c" addicted ;D )
-
Of course I would use one of the most cross platform scripting languages, with deep object orientation, very regular syntax, nice data structures and control flow primitives, auto completion, and mostly self-explaining commands.
I'm obviously talking about PowerShell.
(ducks and runs, but it's quite good TBH).
-
function foo() how many arguments does it need?!? undefined
{
local fid="foo" <------------- isn't there a feature that automatically reports the function name?
local arg1="$1"
local arg2="$2"
local arg3="$3"
...
check_var "arg1" "$arg1" "$fid"
check_var "arg2" "$arg2" "$fid"
check_var "arg3" "$arg3" "$fid"
...
}
${FUNCNAME[0]} contains the name of the currently executing function. The other indices [1], [2], etc. are all the function names in the call stack.
-
PowerShell.
this (https://github.com/PowerShell/PowerShell/) one? :o :o :o
-
Here's a short example without using external calls for comparing a three-part version number, with no more than 3 digits in each part. It uses the same technique as yours by converting the version number to a big integer. Bash is using 64-bit signed integers (at least on my system), so there is room in this method before it breaks.
You could also just do the comparisons for major, minor, and sub-minor separately, which would avoid the one-big-integer conversion problems.
#!/bin/bash
version_ok() {
local ver_cur="${1:?ver_cur not set}"
local ver_min="${2:?ver_min not set}"
local ver_max="${3:?ver_max not set}"
local ver_cur_int
local ver_min_int
local ver_max_int
# extract 3 integer fields separated by ".", and convert
# to one big integer.
#
[[ $ver_cur =~ ^([0-9]*)\.([0-9]*)\.([0-9]*)$ ]] || {
echo "bad version format: $ver_cur" 1>&2
exit 1
}
ver_cur_int=$(( ${BASH_REMATCH[1]}*1000000 +
${BASH_REMATCH[2]}*1000 +
${BASH_REMATCH[3]} ))
echo debug: ver_cur_int $ver_cur_int 1>&2
[[ $ver_min =~ ^([0-9]*)\.([0-9]*)\.([0-9]*)$ ]] || {
echo "bad version format: $ver_min" 1>&2
exit 1
}
ver_min_int=$(( ${BASH_REMATCH[1]}*1000000 +
${BASH_REMATCH[2]}*1000 +
${BASH_REMATCH[3]} ))
echo debug: ver_min_int $ver_min_int 1>&2
[[ $ver_max =~ ^([0-9]*)\.([0-9]*)\.([0-9]*)$ ]] || {
echo "bad version format: $ver_max" 1>&2
exit 1
}
ver_max_int=$(( ${BASH_REMATCH[1]}*1000000 +
${BASH_REMATCH[2]}*1000 +
${BASH_REMATCH[3]} ))
echo debug: ver_max_int $ver_max_int 1>&1
# check for out of range
#
(( ver_cur_int > ver_max_int )) && return -1
(( ver_cur_int < ver_min_int )) && return -1
# version is ok
#
return 0
}
if version_ok 2.5.6 1.2.3 2.8.9; then
echo version is ok
else
echo version is not ok
fi
exit
EDIT: Oops, fixed the RE. I forgot the "*" so the previous code would not accept more than one digit. The dangers of posting code without thorough real-world testing. It should probably also check that each result was not more than 999 since there is only 3 digits allocated in the big int. But the idea was really just to show how some of the internal features of bash could be used.
-
Sorry, fixed a problem with the previous example. See code and the EDIT comment.
-
Lua.
-
PowerShell.
this (https://github.com/PowerShell/PowerShell/) one? :o :o :o
Yes.
TBH, my comment was written with my tongue not firmly planted but definitely leaning a bit on my cheek.
All I have said about PS is, in fact, true, and I even forgot to add the MIT licensing!
As a scripting language it is quite powerful, in conjunction with the .NET Core framework.
Very verbose though, but that makes it less cryptic than others.
I'm not really an expert, I just use it from time to time to automate tasks.
As an example, here is a Wake on LAN cmdlet written in PS, once it's been Import-Module'd, it appears as any other command and responds to most standard options (e.g. -help, autocompletes -mac and -ip etc. etc.):
function Send-WOL {
<#
.SYNOPSIS
Send a WOL packet to a broadcast address
.PARAMETER mac
The MAC address of the device that need to wake up
.PARAMETER ip
The IP address where the WOL packet will be sent to
.EXAMPLE
Send-WOL -mac 00:11:32:21:2D:11 -ip 192.168.8.255
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $True, Position = 1)]
[string]$mac,
[string]$ip = "255.255.255.255",
[int]$port = 9
)
$broadcast = [Net.IPAddress]::Parse($ip)
$computers = @{ redacted = "ff-ff-ff-ff-ff-ff"; missing = "ff-ff-00-00-00-00" }
if ($null -ne $computers[$mac]) {
$mac=$computers[$mac]
}
$mac=(($mac.replace(":","")).replace("-","")).replace(".","")
$target=0,2,4,6,8,10 | % {[convert]::ToByte($mac.substring($_,2),16)}
$packet = (,[byte]255 * 6) + ($target * 16)
Write-Host 'Sending to MAC:' $mac ' IP:' $broadcast
$UDPclient = new-Object System.Net.Sockets.UdpClient
$UDPclient.Connect($broadcast,$port)
[void]$UDPclient.Send($packet, 102)
}
It does not use anything outside PS and .NET Core, and works the same on both Linux and Windows.
Say what you want about Microsoft, .NET bloat (Core is not excessively large), yadda yadda, but it's not that bad.
Of course, I quite doubt it'll work on many of your vintage and interesting machines - I would not venture in even trying a recompile.
-
I know you said "(excluding big elephants like Python, which I like, but it consumes too many resources)", but for non-bash scripting I generally use:
- python
And more recently:
- go (aka golang)
Either are much nicer than bash for scripts of any complexity and they are very good with portability between Windows and Linux. As a portability side note about bash: I've found that the bash interpreter that can be optionally installed with Git for Windows works far better than I ever expected it to at running my Linux scripts. I'd estimate that 80% or more of them run with no changes whatsoever.
Python has the long running irritant of v2 vs v3, but over the past year or more I've found it easy to pretty much ignore v2.
-
Python has the long running irritant of v2 vs v3, but over the past year or more I've found it easy to pretty much ignore v2.
yup, Gentoo Portage is based on Python
It has been a pain since 2005 for me ;D
-
As for "how heavy" are various languages, interpreted as the time taken to start up and run a trivial script and quit, 100 times...
Just the results, in order by wall time:
0.143s awk
0.158s bash
0.162s ksh
0.278s zsh
0.329s perl
0.344s tcsh
0.530s lua
1.957s tcl
2.519s emacs lisp
3.009s python
4.568s ruby
4.979s javascript
15.207s C (compiled each time)
I'd rather use perl than tcl or any of the shells. It has proper syntax and variables and is not based on hairy text interpolation and re-parsing. It was explicitly designed to be a shell replacement for system admins. Where perl falls down is if you want data structures more complex than a simple array or dictionary e.g. nested data structures, trees, graphs etc. The syntax can also be cryptic.
Lua is almost as fast starting up (and much much faster than python or ruby) and is better for writing "real" programs with data structures, OO code than perl. It has more conventional syntax. It doesn't have as much built in for doing system kinds of things but there is os.execute() and io.popen().
Emacs turns out to be surprisingly fast to start up, beating python, ruby, and javascript and surprisingly not far behind tcl. Lisp is a real programming language, suitable for writing very large programs if you're so inclined. The syntax is not for everyone, but personally I find it better than python.
Javascript (node.js) is not all that much slower to start up than python or ruby, has the most C-like syntax if you like that kind of thing, has a huge ecosystem of libraries (similar to python I guess), and so much time and money has been put into the v8 engine that if you want to write complex algorithms then it's by far the fastest language in the above list, coming close to C++ / Java / C#.
Tests run on an M1 Mac.
Details:
bash
Mac-mini:programs bruce$ time (for x in `seq 100`; do bash -c 'echo $((3*14))' >out;done)
real 0m0.158s
user 0m0.042s
sys 0m0.089s
tcsh
Mac-mini:programs bruce$ time (for x in `seq 100`; do tcsh -c '@ r = (3 * 14);echo $r' >out;done)
real 0m0.344s
user 0m0.068s
sys 0m0.188s
python
Mac-mini:programs bruce$ time (for x in `seq 100`; do python3.10 -c 'print(3*14)' >out;done)
real 0m3.009s
user 0m1.875s
sys 0m0.559s
perl
Mac-mini:programs bruce$ time (for x in `seq 100`; do perl -e 'print 3*14,"\n"' >out;done)
real 0m0.329s
user 0m0.076s
sys 0m0.140s
ruby
Mac-mini:programs bruce$ time (for x in `seq 100`; do ruby -e 'print 3*14,"\n"' >out;done)
real 0m4.568s
user 0m3.269s
sys 0m0.964s
awk
Mac-mini:programs bruce$ time (for x in `seq 100`; do awk 'BEGIN {print 3*14}' </dev/null >out;done)
real 0m0.143s
user 0m0.033s
sys 0m0.081s
lua
Mac-mini:programs bruce$ time (for x in `seq 100`; do lua -e 'print(3*14)' >out;done)
real 0m0.530s
user 0m0.176s
sys 0m0.150s
javascript
Mac-mini:programs bruce$ time (for x in `seq 100`; do node -e 'console.log(3*14)' >out;done)
real 0m4.979s
user 0m3.459s
sys 0m0.871s
tcl
Mac-mini:programs bruce$ time (for x in `seq 100`; do echo 'puts [expr 3*14]' | tclsh8.6 >out;done)
real 0m1.957s
user 0m1.057s
sys 0m0.383s
emacs
Mac-mini:programs bruce$ time (for x in `seq 100`; do emacs --batch --eval '(print (* 3 14))' >out;done)
real 0m2.519s
user 0m1.340s
sys 0m0.662s
C
Mac-mini:programs bruce$ time (for x in `seq 100`; do ((echo '#include <stdio.h>';echo 'int main(){printf("%d\n",3*14);}') | gcc -xc -;./a.out) >out;done)
real 0m15.207s
user 0m3.927s
sys 0m1.616s
-
Interesting results. You might also want to give tcc a try on your Mac-mini.
Tcc is an ANSI C compiler that is optimized for fast compilation and it can be used in the same style as bash, perl, and other interpreters by including the following at the top:
#!/bin/tcc -run
<your C program here...>
You don't need the intermediate step of generating an executable.
A quick comparison between gcc and tcc on my system:
piper$ time (for x in `seq 100`; do ((echo '#include <stdio.h>';echo 'int main(){printf("%d\n",3*14);}') | gcc -xc -;./a.out) >out;done)
real 0m2.536s
user 0m1.866s
sys 0m0.705s
piper$ time (for x in `seq 100`; do ((echo '#include <stdio.h>';echo 'int main(){printf("%d\n",3*14);}') | tcc -run -) >out;done)
real 0m0.223s
user 0m0.174s
sys 0m0.069s
-
Interesting results. You might also want to give tcc a try on your Mac-mini.
tcc certainly looks a lot faster for this usage. Unfortunately HomeBrew informs me that "tcc: This formula either does not compile or function as expected on macOS versions newer than Catalina due to an upstream incompatibility."
If "newer than Catalina" includes Catalina, then it might be that tcc only generates 32 bit x86 code as Catalina was the first version to require x86_64 applications only.
Note that MacOS is very slow to run a newly-compiled application the first time, and so tcc might not make much difference. Note that only about 55ms out of the 152ms total time for each run is made up of system+user time, whereas with the others system+user add to something close to the real time.
I think the OS might well be calculating a fingerprint of the app and checking it against an internet database of malicious programs before running it the first time after it is compiled. If I 'cp a.out b.out' and then run the same test, compiling to a.out but then running the saved b.out then the time reduces from 15.2s to 4.4s (faster than ruby and javascript).
My RISC-V VisionFive 2 (with Ubuntu) runs the C test faster than the M1 Mac, in 13.7s, with system+user adding to 13.6. The b.out trick makes no difference at all. The perl and python tests (to pick just two) take just over twice as long as the Mac.
-
It was explicitly designed to be a shell replacement for system admins.
No. It was designed as a script language aimed at processing text and generating reports. Never aimed to be a shell replacement, because the existing shells did their job very well at the time and there was no need to replace them.
Where perl falls down is if you want data structures more complex than a simple array or dictionary e.g. nested data structures, trees, graphs etc.
No. It's very powerful and simple once you get the concept and understand how the nested structures are actually organized.
The syntax can also be cryptic.
Only if you want it to.
p.s. bash vs dash (64-bit x86 linux):
$ time (for x in `seq 1000`; do bash -c 'echo $((3*14))' >out;done)
real 0m1.177s
user 0m0.830s
sys 0m0.388s
$ time (for x in `seq 1000`; do dash -c 'echo $((3*14))' >out;done)
real 0m0.707s
user 0m0.462s
sys 0m0.287s
-
Interesting results. You might also want to give tcc a try on your Mac-mini.
Tcc is an ANSI C compiler that is optimized for fast compilation and it can be used in the same style as bash, perl, and other interpreters by including the following at the top:
#!/bin/tcc -run
<your C program here...>
You don't need the intermediate step of generating an executable.
Wow, never heard of it. Live and learn. It's an interesting thing, however, what would a real use case be for it? Hmm...
-
Bash is good, but it's too error prone.
What would you use to replace it?
Well, first of all, what's your target? What range of *nix (only Linux or also other Unixes?) and what type of h/w (beefy server or small SBC) should your script run on?
-
Well, first of all, what's your target? What range of *nix (only Linux or also other Unixes?)
only GNU/Linux, I have no plans for {net , open , free, ...}-BSD
and what type of h/w (beefy server or small SBC) should your script run on?
from embedded GNU/Linux SBC boards(1) to small (for today standards) GNU/Linux servers(2).
(1)
the worst in terms of resources: PPC40x@133Mhz, 32MB ram, 5Mbyte/sec storage
the best in terms of resources: ARM-A53@1100Mhz, 1GB ram, 20Mbyte/sec storage
(2)
the worst in terms of resources: MIPS32R2/BE@680Mhz, 128MB ram, 10Mbyte/sec storage
the best in terms of resources: Intel-CoreDuo2@2200Mhz, 8GB ram, 50Mbyte/sec storage
-
Now, the question is, WHY do you want to replace bash with anything? It makes zero sense to do it for boot scripts, since they are executed only once. In terms of performance, you generally need something faster only if the program is to be executed at least a few hundreds or thousands times per second.
In terms of convenience, yes, bash and other shell languages aren't particularly well suited for big and/or complex programs, but then the question has to be "what language should I use for a given task?" rather than what the thread title says.
-
Where perl falls down is if you want data structures more complex than a simple array or dictionary e.g. nested data structures, trees, graphs etc.
No. It's very powerful and simple once you get the concept and understand how the nested structures are actually organized.
I first user perl professionally to do system configuration and migration tasks at Unisys for the NZ Inland Revenue "Server Redeployment Project" in 1997, where old SunOS servers turned up at my desk each day, I upgraded the RAM and disks in them, installed and configured Solaris and other software, and shipped them back to their appropriate offices.
There have been few days since when I haven't used existing perl scripts or one-liners. I've written some very complex stuff in perl. I think I know how complicated data structures work in perl and it is AWFUL compared to Python, Ruby, Javascript, Lua, C, Java, or just about anything else.
Simple hashes of hashes of hashes etc and hierarchical things in general are fine. Generalised graphs are not. They are UG-leeee.
-
Now, the question is, WHY do you want to replace bash with anything? It makes zero sense to do it for boot scripts
For me, boot-bash-scripts are too error prone to maintain and adapt to different needs.
They are not even used once but rather on demand to stop, reconfigure, and reload a service.
/etc/system/init/wifi stop
...
/etc/system/init/wifi start
Also, I want to use *the same language* to write tools that I have to use on a daily basis.
Python-based tools (including "emerge" and a great part of OpenRC) are too CPU-and-ram-hungry.
-
+=dev-lang/lua
+=dev-lua/luafilesystem
+=dev-lang/ruby
+=app-shells/dsh #(distribuited shell, seems interesting for the cluster)
+=app-shells/dash
added in the profile list of the new stage{1,2,3,4} under cooking.
cooking time, on the cluster of five Mac-Mini-i2: 96 hours for { mips32le, ppc32be, hppa2/32bit-be }
next week I will try to learn and experiment with new dev-lang/*, and with the two new shells ;D
-
perl
yup, I have also to support dev-lang/perl (at least on the building machine, whose profile is in use on the squad of Mac-Mini-i2) because it's used by several linux kernel scripts.
To build the kernel linux, you need perl.
In older kernel 2.6.* perl scrips need a patch to be compatible with modern perl interpreters.
(I don't know about kernel 2.4.*)
I am with perl v5.30.3, I think I won't update, and I don't think I will push it onto target-stages, so, /usr/bin/perl5 and the rest of the package (plus tons of dependencies, perl has many on Gentoo) won't be included in the final rootfs, e.g. on my Japanese GNU/Linux PDA, unless strictly required by some application :-//
It will also save precious space on the little built-in nand flash.
-
For me, boot-bash-scripts are too error prone to maintain and adapt to different needs.
They are not even used once but rather on demand to stop, reconfigure, and reload a service.
Take care of startup dependecies! Only limited functionality of the system may be available when rc scripts are executed during system startup. For instance, if /usr is not yet mounted when a particular script is executed, then the script can't use a language interpreter which resides in /usr, and it also can't invoke any tools residing in /usr. Many rc scripts are therefore limited to /bin/sh and only a couple of tools residing in /bin or /sbin.
-
There have been few days since when I haven't used existing perl scripts or one-liners. I've written some very complex stuff in perl. I think I know how complicated data structures work in perl and it is AWFUL compared to Python, Ruby, Javascript, Lua, C, Java, or just about anything else.
Simple hashes of hashes of hashes etc and hierarchical things in general are fine. Generalised graphs are not. They are UG-leeee.
Well I have had quite an extensive experience with perl, including writing big programs, too. It's a lovely language, exceptionally easy to understand and use, and this applies to its data structures too. Basically, they are as simple or as complex as the programmer makes them. The language doesn't really enforce anything.
Comparison with C is weird for me. In C, it may become really tedious, unless you involve libs that do all the low-level heavylifting.
-
For me, boot-bash-scripts are too error prone to maintain and adapt to different needs.
They are not even used once but rather on demand to stop, reconfigure, and reload a service.
Now I'm starting to smell a little bit of something being not quite right there, such as trying to solve a problem already solved.
Are you writing your own init scripts? Are you writing them for your own software?
-
Are you writing your own init scripts?
Are you writing them for your own software?
I thought it was clear that I wrote my own init scripts and my own tools in bash and I want to use something different for the next gen.
-
Take care of startup dependecies!
yup, each bash module directly reports its needs
e.g. sshd depends on network
there is an engine, written in C and directly called by /sbin/init (it's a build-in thread), that elaborates all the modules and outputs (pipe) a list based on priority and dependency
rte
mount-rootfs
kernel-specific
kernel-modules-dep
mount-others
mount-swap
system-clean
memory-clean
clock-simple
machine-identify
hostname
hostname-server
machine-specific
wlan-mesh-batman
networking-bridge
networking-loopback
networking-gateway
networking-ipforwarding
networking-syn
networking-snort
env-shared-libraries
http-tini-server
sftp-server
ssh-server
tftp-server
iperf-server
hostname-discovery-server
ttyS-init
uart-init
crypto-init
(e.g. on my router)
then it invokes a scheduler (written in bash) that reads the list from the pipe, and launches all the services found in etc/system/init/* (modules there are written in bash), one by one, without competition.
It's very clean, faster than OpenRC, and simple. But ... I think it can be improved with scripts written in Lua.
As well as my patch-manager and all the other big and complex bash scripts can be improved if written in Lua.
-
Take care of startup dependecies!
yup, each bash module directly reports its needs
e.g. sshd depends on network
there is an engine, written in C and directly called by /sbin/init (it's a build-in thread), that elaborates all the modules and outputs (pipe) a list based on priority and dependency
What I mean is, if all scripts are written in (say) Lua, then you also need to have Lua (and its prerequisites) installed in the initramfs.
[ Fortunately, Lua is still pretty lightweighted, compared to Python, Perl, V8, and others. ]
-
If you have to use bash or sh for scripting, have a look at ShellCheck:
https://www.shellcheck.net/ (https://www.shellcheck.net/)
It's a linter for shell scripts and it can catch a lot of common problems.
-
Take care of startup dependecies!
yup, each bash module directly reports its needs
e.g. sshd depends on network
there is an engine, written in C and directly called by /sbin/init (it's a build-in thread), that elaborates all the modules and outputs (pipe) a list based on priority and dependency
What I mean is, if all scripts are written in (say) Lua, then you also need to have Lua (and its prerequisites) installed in the initramfs.
[ Fortunately, Lua is still pretty lightweighted, compared to Python, Perl, V8, and others. ]
Yep. Lua is very small. It's more or less about 300KB of binary code.
I have a "minimal" Python 3.10 "distribution". Takes about 370MB, so 3 orders of magnitude more. ;D
-
-> can override functions without a warning
Use
function funcname () {
...body...
}
declare -rf funcname
and function funcname can be referenced and used, but not redefined.
Note that function and variable namespaces are separate: you can also have a variable funcname. You may therefore wish to add
declare -r funcname=funcname
-> no actual return value
Use the 'first argument specifies result variable name' pattern:
function funcname () {
[[ $# -ge 2 ]] || return 1
declare -n retval="$1"
shift 1
...body...
retval=value-or-expression
}
declare -rf funcname
declare -r funcname=funcname
and you can modify the named variable in the caller scope. Granted, this isn't "return value" per se, but passing a variable by reference, allowing modifying the variable in the caller scope.
In Bash, and many other shells, the return value is always status; zero for success, nonzero values indicating errors.
Exit status itself is always accessible as $?.
-> the number of arguments is not checked
Just like in C main(), you need to do it yourself. $# is the number of positional parameters ($1, $2, ..., also ${1}, ${2}, ...), NOT including the function name itself ($0 or ${0}. In the example before, the line
[[ $# -ge 2 ]] || return 1
means that unless there are at least 2 positional parameters given, the function returns immediately with exit status 1.
-> arguments data-type is not checked
Not automatically, no. Technically, all shell variables and environment variables are strings, so there are no data types.
You can always define a function that assigns the value of an integer expression to a variable, or returns nonzero if it is not possible. For example, if you want to evaluate any integer expressions,
function inteval () {
[[ -n "$1" && $# -ge 2 ]] || return 2
declare -n retval="$1"
shift 1
( retval=$(($*)) ) 2>/dev/null || return 1
retval=$(($*))
return $?
}
For example, inteval X 2*3+4 returns with success, with X=10.
To verify a specific variable or value is an representable integer,
function isint () {
[[ $# -eq 1 && $1 =~ ^[-+]?[0-9]+ ]] || return 1
( tmp=$(($1)) && [[ "$tmp" == "${1#+}" ]] ) 2>/dev/null || return 2
return 0
}
For example, isint -254 returns with success, but isint 0x254 with failure.
Similarly, a large enough integer in magnitude will overflow and therefore return with failure.
the integer data type is simulated by external tools
No, integer arithmetic is implemented within Bash itself, and is quite fast. All assignments of form VAR=$((expression)) (or VAR=$[expression]) will either assign an integer value of the expression to VAR, or fail with an interpreter error.
I do prefer the $[expression] form over $((expression)), because there is a risk of confusing the latter with $( (statement) ), which is replaced with the output of statement.
It is true that Bash does not warn about integer arithmetic overflow. If you try e.g.
X=$[ 1208925819614629174706176 ]
then X will be assigned a zero, because the right hand evaluates to 280 which wraps to zero. Then again, C behaves the same.
-> compare two numbers is ugly, and too error prone
Ugly is in the eye of the beholder.
As to error prone, not so, when one uses the -lt, -le, -eq, -ne, -ge, and -gt arithmetic comparison operators in [ expression and [[ expression ]] expressions. <, <=, ==, !=, >=, and > are lexicographic comparisons in [[ expression ]], not arithmetic comparisons. =~ is a POSIX extended regular expression (string) match operator.
-
@Nominal Animal: a nice set of suggestions, especially the $[expression] syntax which I was unaware of. I'll be trying it out to see if I like it better than $((expression)). I'm guessing that I'll be sticking with $((expression)) because the [] brackets make my brain think of of arrays at first glance (not sure why that doesn't happen with if [ ... ]). But I'll give it a go.
-
@Nominal Animal: a nice set of suggestions. especially the $[expression] syntax which I was unaware of. I'll be trying it out to see if I like it better than $((expression)). I'm guessing that I'll be sticking with $((expression)) because the [] brackets make my brain think of of arrays at first glance (not sure why that doesn't happen with if [ ... ]). But I'll give it a go.
Bash developers do intend to eliminate $[expression] at some point, with $((expression)) the preferred way. POSIX shell spec also uses the latter, so consider me in the wrong here. (I also object to the standard backslash escape mechanisms, especially \\ as an escaped \. I much prefer '\/' for '\', '\,' for '"', '\.' for ':', and so on.)
For Dash users:- Autoremoved temporary directory works:
readonly Work="$(mktemp -d)" || exit 1
trap "rm -rf '$Work'" EXIT
Just like in Bash, the the directory to be removed is substituted from the variable at the time the trap is set (because of the outer double quotes), and not at the time the trap triggers. This means that even if you omit readonly, doing e.g. Work="/" does not change anything: the original temporary directory is deleted regardless.
- Default value assignment works:
VAR="${VAR:-default}"
If VAR is unset or empty, it will be set to default.
Similarly, ${#VAR} is the length (number of bytes in the C locale) of the value of VAR, ${VAR%suffix} and ${VAR%%suffix} remove the shortest or the longest (shell/glob pattern) suffix, and ${VAR#prefix} and ${VAR##prefix} remove the shortest or longest (shell/glob pattern) prefix.
- Arithmetic substitution via $((expression)) works.
Variable names do not need to be preceded by $: $((x*y)) calculates the product of $x and $y, if both are integers.
- Logical expressions use [ expression ] syntax (not (( expression ))).
<, =, !=, and > are lexicographic/string comparison operators.
For arithmetic comparisons, use -lt, -le, -eq, -ne, -ge, and -gt.
The -n "string" (true if nonempty string) and -z "string" (true if empty string), as well as the various test utility file/directory flags are also supported.
- Function definitions do not use a keyword, they start with the function name:
funcname () {
...body...
}
- Function definitions cannot be made immutable.
readonly funcname
only sets the variable funcname readonly. Function names are in a separate namespace, and at least I haven't found any way to make Dash spit out an error when trying to redefine an already defined function.
- Variables are scoped. Therefore, a function evalint that evaluates an arithmetic expression saving it in the variable named in the first positional parameter, can be written as
evalint () {
[ $# -ge 2 -a -n "$1" ] || return 2
local name="$1"
shift 1
[ "$name" = "${name##*[^_0-9A-Za-z]}" ] || return 2
local value="$*"
( value=$(($value)) ) 2>/dev/null || return 1
value=$(($value))
eval "$name=$value"
return 0
}
which will return success ($? zero) if successful and variable set, false (nonzero $?) otherwise.
Note that while the eval looks scary above, name has already verified to contain only underscores and alphanumerics, and value is an arithmetic result. So, it is actually safe, barring internal Dash bugs.
For example,
evalint x 2*3+4
will be successful and set x to 10, but
evalint x 2.2+1
will fail.
For what it is worth, Bash startup latency on my system is about 7 milliseconds, whereas Dash startup latency is just 4 milliseconds. If you run both under strace, you'll see Dash does much less initial work.
So, my answer would be 'Dash'. :D
-
If you're looking for a shell scripting language that is less error-prone than Bash, but don't want to use a language like Python due to resource constraints, one option to consider is Fish (Friendly Interactive Shell). Fish is designed to be more user-friendly and less error-prone than Bash, with features like syntax highlighting and automatic suggestions. It's also more powerful than traditional Unix shells, and includes support for functions, variables, and other programming constructs.
Another option to consider is Zsh (Z Shell), which is similar to Bash but includes additional features like tab completion and spelling correction. Zsh is also highly customizable, with support for plugins and themes.
-
Having worked with many languages over the last 20 years and havng had to use Lua professionally in the last year, I'd say "anything but Lua". If bash is working for you, then stick with that.
-
Having worked with many languages over the last 20 years and havng had to use Lua professionally in the last year, I'd say "anything but Lua". If bash is working for you, then stick with that.
Care to expand on that?
-
Having worked with many languages over the last 20 years and havng had to use Lua professionally in the last year, I'd say "anything but Lua". If bash is working for you, then stick with that.
Your lack of non-revisionistic attitude is disconcerting :)
-
Bash error-prone? You could say the same for each existing programming language.
Bash does its job fine, anywhere just write and run, I've made lots of complex bash scripts for Openwrt.
If you want top performance, C is the way to go, but for checking something every second it'll do just fine.
Use sleep command, otherwise it'll loop as fast as it cans, wasting cpu power.
It seems what you really need is to learn/practice it!
-
It seems what you really need is to learn/practice it!
Debugging Bash scripts with poor built-in debugging features? Well ... there are better hobbies :D
-
Bah error-prone? You could say the same for each existing programming language.
Some kinds of programming languages are more problematic than others.
Those based on textual substitution, repeated parsing, levels of quoting of special characters etc are especially prone to programming errors and often security risks as a result.
These include most shell scripting languages, TCL, SQL code assembled at runtime rather than precompiled with variables bound at runtime, any language with "eval", and the C preprocessor.
While algorithms using text substitution have almost as long a history in formal computer science as lambda calculus and Turing machines, there is a reason why we don't usually use any of them as-is for modern programming.
-
You should deal with scripts in language that is used by your company. Using something unpopular gain you more issues when code will be changed / maintained by some1 else.
Compiled language are fastest, but think about using c++ or sth like that for maintenance - even when u have buildin filesystem library (c++17), boost, stl etc.
Pyton have to much issues with low level entry - thats why its performance is bad - there is a lot of checks undehood - but you can make your own precompiled libraries and make bootlenecks with c++ etc.
My company using perl for scripting and its ok, a lot of libraries etc. Try out few and choose by yourself, everyone should have own opinions based on own experience.
ps. Even in lower lvl languages u can google library - its not a shame to dont know everything. Issue with interpreters cannot be solved - you will notice them when u gain experience, but u will lose more time to finding them than learning somethink lower imo
-
I will cast another vote for perl. I'm not particularly comfortable with it myself, but compared to bash I would pick it every day. It's more powerful than bash and has fever foot-guns. I'm not very fond of the syntax, but for one-liners I get that it is compact. It's old, established, not going away, and it is fast in start-up.
-
I thought the main thing about a "scripting language" is that it gave you easy access to the same commands that a user would use at the CLI prompt - built-in shell commands and ability to run/script/scrape other user programs.
I didn't think that languages like Lua did that (or not easily, anyway.) Certainly standardish C (tcc) doesn't.
-
I thought the main thing about a "scripting language" is that it gave you easy access to the same commands that a user would use at the CLI prompt - built-in shell commands and ability to run/script/scrape other user programs.
I didn't think that languages like Lua did that (or not easily, anyway.) Certainly standardish C (tcc) doesn't.
Lua has the same system() and popen() kind of calls as any other language, including Perl and C.
The hallmark of the "scripting language" is really, I think, just making it easy to create/handle the arrays for executing commands the string handling to parse/analyse their output.
-
Lua has the same system() and popen() kind of calls as any other language, including Perl and C.
Well, C would be an incredibly poor choice of scripting language (compiled nature, cumbersome and limited string processing, cumbersome execution of external commands), so that's not much of an argument :-DD. I'd say Perl is in between shell scripts, which make executing external commands trivial, and languages like Python or Lua, which require you to use a more cumbersome syntax like system() and popen. For example I imagine this would be a bit longer in Lua or Python (get list of packages from the dpkg command and print them in a different format):
for my $package_line (`dpkg -l`) {
my ($status, $name, $version, $arch, $description) = split(/ +/, $package_line);
print("$name $version\n");
}
-
Lua has the same system() and popen() kind of calls as any other language, including Perl and C.
Well, C would be an incredibly poor choice of scripting language (compiled nature, cumbersome and limited string processing, cumbersome execution of external commands), so that's not much of an argument :-DD. I'd say Perl is in between shell scripts, which make executing external commands trivial, and languages like Python or Lua, which require you to use a more cumbersome syntax like system() and popen. For example I imagine this would be a bit longer in Lua or Python (get list of packages from the dpkg command and print them in a different format):
for my $package_line (`dpkg -l`) {
my ($status, $name, $version, $arch, $description) = split(/ +/, $package_line);
print("$name $version\n");
}
Well, sure, and that's why I still use Perl every day, not Python.
But
1) C is only a library away from being ok, and you only have to write that library once.
2) even perl looks pretty cumbersome compared to dpkg -l | awk '{print $2, $3}'
-
1) C is only a library away from being ok, and you only have to write that library once.
A library that will add syntax for better string handling? So one can treat strings as a scalar type which can be copied by assignment, concatenated using the + operator, etc? I guess you mean C++ that with it's operator overloading has sort of reasonable string functions. A library that removes the need for compilation? Sure, you could try to make C a slightly less horrible choice with some duck tape, but why would you want to?
2) even perl looks pretty cumbersome compared to dpkg -l | awk '{print $2, $3}'
Agreed, that's why I'd say that shell scripts are the first choice here, Perl second, Ruby maybe third, then Python / Lua and other general-purpose scripting languages and C somewhere near the end just above Java and Haskell.
-
A library that will add syntax for better string handling? So one can treat strings as a scalar type which can be copied by assignment, concatenated using the + operator, etc?
my-C can do it, if your source complains level >=2 then safe_string_t is a native built-in datatype :D
safe_string_t + safe_string_t :: concatenate
safe_string_t - safe_string_t :: difference
safe_string_t * safe_string_t :: NULL
safe_string_t / safe_string_t :: NULL
safe_string_t % safe_string_t :: NULL
so, I'm ***really*** tempted to recycle part of my-c code to write an interpreter.
but why would you want to?
this is the current question that prevents me from doing it :o :o :o
I can't answer **why?** with anything but "because I'd like to"
meanwhile I am experimenting with Lua :-//
-
Lua is fine for this, it's robust, lightweight and won't take any more of your development time.
Of course nothing prevents you from designing your own interpreted language.
I did that a (long) time ago, at the time Lua actually existed but I hadn't heard about it, it was still pretty "confidential", Python did exist too but it was a major pain at the time to include the interpreter inside another app without nasty side-effects, and I already had the basis (as you do) to develop a programming language (my own parser, analyzer for algebraic expressions, etc.)
It turned out pretty good, but these days I don't think I would go that route unless I had a very good reason for doing so. Lua has become robust, useful, well documented, is easy to integrate inside any program...
Apart from this, two great features of Lua are: the tables as a first-class type (allows you to implement almost any data structure very efficiently, and object-oriented stuff on top of that), and the built-in regex handling, which is very powerful as well. Just implementing and validating both yourself would take a pretty long time.
-
2) even perl looks pretty cumbersome compared to dpkg -l | awk '{print $2, $3}'
Agreed, that's why I'd say that shell scripts are the first choice here, Perl second, Ruby maybe third, then Python / Lua and other general-purpose scripting languages and C somewhere near the end just above Java and Haskell.
I fully agree with your ordering, but with Lua definitely ahead of Python at least for bigger scripts, I'm not sure about one-liners.
-
step1: convert some booting scripts from Bash scripting to Lua scripting
step2: convert **ALL** booting scripts
step3: convert some utilities scripts (e.g. patch manager) from Bash scripting to Lua scripting
step4: convert **ALL** utilities scripts
and lua-term (https://github.com/hoelzro/lua-term) seems interesting: adds terminal functions for Lua :o :o :o :o
-
Found this (https://github.com/LuaDist/toluapp) interesting tool to integrate C/C++ code with Lua.
So, I can recycle some C-libraries, and write the rest in Lua!
-
There are also cffi (https://github.com/q66/cffi-lua) and cdecl (https://github.com/koreader/ffi-cdecl)
-
Found this (https://github.com/LuaDist/toluapp) interesting tool to integrate C/C++ code with Lua.
So, I can recycle some C-libraries, and write the rest in Lua!
Calling C code from Lua is part of Lua itself, read the docs. You don't need any third-party tool to do this.
-
Found this (https://github.com/LuaDist/toluapp) interesting tool to integrate C/C++ code with Lua.
So, I can recycle some C-libraries, and write the rest in Lua!
Calling C code from Lua is part of Lua itself, read the docs. You don't need any third-party tool to do this.
Plain Lua can only call C functions with the signature
int function_name(lua_State *L)
It cannot call arbitrary C functions.
With CFFI you can call (almost) arbitrary C functions directly, without implementing a wrapper. CFFI has some overhead, though.
-
Calling C code from Lua is part of Lua itself, read the docs. You don't need any third-party tool to do this.
Those tools *help* integration.
-
Found this (https://github.com/LuaDist/toluapp) interesting tool to integrate C/C++ code with Lua.
So, I can recycle some C-libraries, and write the rest in Lua!
Calling C code from Lua is part of Lua itself, read the docs. You don't need any third-party tool to do this.
Plain Lua can only call C functions with the signature
int function_name(lua_State *L)
It cannot call arbitrary C functions.
With CFFI you can call (almost) arbitrary C functions directly, without implementing a wrapper. CFFI has some overhead, though.
Yes. I usually favor going for as few third-party stuff as possible and having complete control about the function interfaces, but to each their own obviously.
Using this helper will get some benefits of course but will also have some overhead, make you have less control and possibly give you headaches when you'll want to update Lua itself.
But this "as few third-party stuff as possible" policy is obviously not very popular, so, just mentioning that in case.
-
I’ve written a lot of tcl in my career and find it a pleasant environment when I need something more than 20 lines or so of bash.
Lately, I’ve been playing with babashka, but that only makes sense on a team already using Clojure.
https://github.com/babashka/babashka
-
I’ve written a lot of tcl in my career and find it a pleasant environment when I need something more than 20 lines or so of bash.
Lately, I’ve been playing with babashka, but that only makes sense on a team already using Clojure.
https://github.com/babashka/babashka
WOW, ***very*** interesting!!!
# app --search babashka
Searching...
Applications found : 0
Overlay created to start experimenting!
# app --search babashka
Searching...
idp: dev-lang/babashka
(source = git, so it's =dev-lang/babashka-9999 )
-
oh, nmap massively uses lua scripts :o :o :o :o
interest+=1000
-
But this "as few third-party stuff as possible" policy is obviously not very popular, so, just mentioning that in case.
Depends on complexity. Re-implementing the functionality of an existing mature 3rd party subsystem consisting of of say 100000+ LOC is not done in a day.
[ Edit: The actual strength of Python is IMO rather not the lnaguage, but the existance of bindings for a large amount of 3rd party software. ]
-
Depends on complexity. Re-implementing the functionality of an existing mature 3rd party subsystem consisting of of say 100000+ LOC is not done in a day.
Yup. Precisely :D
the idea is: quickly move and then(1) perfection it by get rid of external support.
(1) definitely in >=2024
-
But this "as few third-party stuff as possible" policy is obviously not very popular, so, just mentioning that in case.
Depends on complexity. Re-implementing the functionality of an existing mature 3rd party subsystem consisting of of say 100000+ LOC is not done in a day.
Of course, except that you don't always (or even: rarely) need all the shebang that comes with it and certainly don't need to rewrite 100k LOC of code to implement whatever you need implemented.
And now you're stuck with an enormous thing that you (basically) have no control over.
That's the whole issue with software bloat in general.
Here, sure I don't know what the OP wants to do, but you don't need to write 100k LOC to wrap a few functions in Lua C functions. A few tens of LOCs is probably all you need.
But funnily enough, at some point, people start convincing themselves that they need a several-GB worth of tools with thousands of libraries available just to calculate the average of a few numbers.
Of course that depends on complexity.
But people in general tend to largely overestimate the complexity ot the tasks they have at hand.
Not everyone is Wirth.
Some related fun: https://gwern.net/doc/cs/2005-09-30-smith-whyihateframeworks.html
;D
PS.: As the goal was to replace Bash, I don't see a single functionallity that would be missing from Bash when using Lua and that would even require interfacing with external C functions directly. Can you do that with Bash? OTOH you can call any executable from Lua just like with Bash, using the os.execute() function that is part of the base Lua libraries. Look up all 'os' functions available, that should get you covered for the most part.
-
The only issue I have found bash to be frustrating with is the LFS system and delimitation of string data and lists.
Consider
# for i in $(ls /somefolder/); do echo $i; done
This is fine. However .
# for i in $(ls /somefolder/); do mv $i "other folder/"; done
Won't.
Even when you try to fix this with the LFS variable in a script you find it still gets ugly. Some times you end up having to use the lowever level streaming features like the "read" command.
Instead, I stipulate that anyone putting a space or any other common delimiter in a filename, gets at least 6 lashes.
-
for i in $(ls /somefolder/); do echo $i; done
You don't want to do that pattern, for a number of reasons. Most important one is that ls is the wrong tool here.
Instead of ls, either use glob patterns, for example (POSIX sh compatible, including dash)
for i in /somefolder/* ; do [ -e "$i" ] || continue ; echo "$i" ; done
or all into a Bash array,
read -d "" -a pathnames < <(find /somefolder/ -mindepth 1 -maxdepth 1 -print0)
Both handle the cases with whitespace (including newlines) in file and directory names correctly.
The third commonly used pattern, using e.g. find and read them in a loop one by one,
find /somefolder/ -mindepth 1 -maxdepth 1 -print0 | while read -d "" i ; do
echo "$i"
done
also works well but is a bit slower than the other methods.
If you use
shopt -s nullglob
early in your Bash script, non-matching glob patterns expand to a empty string instead of the glob pattern, and you can simply use
for i in somefolder/* ; do echo "$i" ; done
which simplifies many per-file loops. Furthermore, you can do
pathnames=(somefolder/*)
to obtain a Bash array of all matching path names, with an empty array if there are no matching path names. (Normally, in the no matches case, the array would have the glob patterns themselves.)
Note the quotes around "$i".
When i="foo bar", $i expands to two tokens (assuming default field separator IFS), but "$i" always expands to a single token.
Similarly with arrays: ${name[@]} is re-split into tokens based on IFS, but "${name[@]}" expands to one token per array element.
(The number of elements, ${#name[@]}, is always an integer, and therefore does not need to be quoted.)
# for i in $(ls /somefolder/); do mv $i "other folder/"; done
Won't.
I often use
for fn in /somefolder/*pattern* ; do echo mv -vi "$fn" "other folder/" ; done
to verify which files I want to move, and then rerun it without the echo to do it for real. It will list the moved files, and verify before overwriting anything.
Of course, a simple
mv -vi /somefolder/*pattern* "other folder/"
does the same thing, if the pattern doesn't match too many files. (In Linux, the limit is so high, though, that most people don't ever hit it.)
Some times you end up having to use the lowever level streaming features like the "read" command.
It's a shell built-in, and not that slow. On my system, a while read -d $"\n" line ; do true ; done < /usr/share/dict/words reads about a hundred thousand lines per second.
Then again, reading the same into a Bash array,
OLDIFS="$IFS" ; IFS=$'\n' ; words=($(< /usr/share/dict/words)) ; IFS="$OLDIFS"
takes only a tenth of a second, which shows how useful Bash arrays are. The downside is that it does not support nul ($'\0') as a separator.
I write quite a lot of Bash scripts, and I test mine with nasty file names having newlines and invalid UTF-8 sequences (my locale is an UTF-8 one). I just get the heebie-jeebies if there is a risk a script may mangle my preciousss data..
-
Bash is good, but it's too error prone.
What would you use to replace it?
(excluding big elephants like Python, which I like, but it consumes too many resources)
Php ... well, it looked a good idea, after 1 year, let me say: never again!
Lua? eLua?
...
Looking for something I can use to replace my startup system scripts (/etc/init.d/*), as well as my system scripts (in /usr/bin/), currently all written in Bash-scripting.
For purely synchronous tasks that don't need much parallelisation PHP is a good choice.
If you can stomach the error handling that is, and don't need modularization (PHP sucks at those).
Otherwise, I'd use nodejs because it enables very powerful parallelisation and modularization without much effort (think async/await and import/require).
And ...
Php ... well, it looked a good idea, after 1 year, let me say: never again!
I know where this is coming from and this is the reason I'll be migrating ALL my scripting/building/tooling scripts from PHP to JavaScript. (see my points above)
-
in the bay area, if you do anything that's not C or C++, its python.
end of story.
-
There is a massive amount of javascript (react and node) being written in the valley.
I'd be surprised if c, c++, and python added together represented even 40% of the code written in SF, the valley, California, the US, or the world.
-
I found it hard to explain and produce examples, as it's not the "first" evaluation of the space and other delimiter containing data that is an issue. You can just "" quote it.
However, every bash evaluation of a quoted string, unquotes it. So if it then passed as an argument to a program in teh script it again loses it's "single argument" nature.
#!/bin/bash
stuff=$1
mplayer $stuff
Then calling it with:
./script.sh "My movies.mpg"
Will usually result in
My: File not found
movies.mpg: file not found.
There are syntactic sugars which I believe can change how base expands or does not expand the quotes, but I never remember them and they are hard to google.
-
Sometimes it can be interesting to have a C language interpreter that does not need to compile the sources. TCC is the answer: https://bellard.org/tcc/ (https://bellard.org/tcc/)
Lua is very lightweight and portable, if that's what you're looking for.
Python without libraries can also be lightweight. And, although the battery-powered language is a bit heavier, for some reason it comes standard in most major Linux distributions.
AWK is another option. I used it a lot to automate some tasks before I got to know more "Heavy" languages like Python.
Another option is TCL: https://www.tcl-lang.org/ (https://www.tcl-lang.org/)
Edit:
Last, but not least. Javascript.
-
TCC is great and can be used as an interpreter, but it compiles so fast anyway that you can even use it in similar ways as an interpreted language while still running compiled code.
It compiles a several KLOCs C file faster than Python takes to initialize. ;D
-
However, every bash evaluation of a quoted string, unquotes it. So if it then passed as an argument to a program in teh script it again loses it's "single argument" nature.
#!/bin/bash
stuff=$1
mplayer $stuff
Yes, that is absolutely true. The only "fix" is to use proper quoting within the scripts everywhere (so stuff="$1" and mplayer "$stuff" in the above example). The nice thing is that after you do do that and it becomes second nature, you know your scripts (like mine) can handle all possible filenames in Linux, even those including newlines or other control characters.
Bash (and POSIX sh) shell script language is a funky beast, and double quotes should be considered syntax for disabling word expansion; similar to how single quotes disable all mangling and expansions.
My typical work scripts begin with
#!/bin/bash
export LANG=C LC_ALL=C
Work="$(mktemp -d)" || exit 1
trap "rm -rf '$Work'" EXIT
where setting LANG and LC_ALL ensures non-ASCII bytes in names are treated as-is and not attempted to parse according to the locale charset, and the last two lines create a temporary work directory that is autoremoved when the script exits (even if killed by a signal or some other error). The quote order in the fourth line means the work directory path is expanded on that line, when the trap is set, so that even if the variable is modified later on, the original work directory gets removed.
If the script deals with globbing files, I insert the shopt -s nullglob before the export line, so that glob patterns that don't match anything, expand to an empty string (instead of the pattern itself as is default).
One really needs to read (most of) the Bash manual (https://www.gnu.org/software/bash/manual/bash.html), and grok its idiosyncracies and inanities, before one can learn how to write reliable Bash scripts. I agree that to many, it is just too much effort compared to how little it is used in real life. I've written quite a lot of Bash code myself, including a bespoke init system for benchmarking compute nodes, so I basically had to spend the time to learn it well.
A good example of the hidden complexity and gotchas is the following Bash snippet, and why it outputs nothing:
X="" ; echo foo | read X ; echo $X
The fixed version that does what one would think is
X="" ; echo foo | ( read X ; echo $X )
but X is still empty afterwards. The recommended version is
X="" ; read X < <( echo foo ) ; echo $X
where X is foo afterwards. A normal user or sysadmin does not need to know this, unless they write a lot of maintenance or init scripts.
(The reason for the behaviour is that right side of pipe, |, is always in a subshell, and changes to variables in a subshell are not propagated to a parent. Parentheses form a subshell, so in the second variant the echo is in the subshell and not the parent. The third form executes the echo foo command in a subshell, and redirects it as the input to the read built-in in the parent shell; this is why the feature was created in the first place. The first < is the input redirection, and the <(...) creates a subshell redirecting its output to a pipe, and in the parent shell evaluates to the path of the other end of the pipe, that is in Linux only valid for that process and not readable by any other processes. echo <(true) typically outputs a descriptor-based path in Linux, /dev/fd/NN; the path would only work for that echo process, and not any other, so there is no risk of leaking the pipe contents.)
-
Yesterday my bash libraries reached 12K lines of code.
Not for hubris talking, but this stuff is unmanageable.
So, I am going ahead for inertia, but I am already moving to Lua.
10% progress :-+
-
Yesterday my bash libraries reached 12K lines of code.
Not for hubris talking, but this stuff is unmanageable.
So, I am going ahead for inertia, but I am already moving to Lua.
10% progress :-+
Hey, I always recommend people use whatever tool that works best for them, as long as the tool is suitable for the problem at hand; and Lua is a widely used scripting language.
While I haven't seen what Lua would look like when used to implement command-line utilities, I don't see any reason why it would not be suitable here.
If nothing else, it is an interesting experiment. I do a lot of stuff just as an experiment myself: it's the best way to find out and learn. :-+
-
Yesterday my bash libraries reached 12K lines of code.
Not for hubris talking, but this stuff is unmanageable.
So, I am going ahead for inertia, but I am already moving to Lua.
10% progress :-+
You truly have earned my respect for this :-+ :-+ well done! I'm doing something similar, moving from PHP to all JavaScript 8)
Hey, I always recommend people use whatever tool that works best for them, as long as the tool is suitable for the problem at hand; and Lua is a widely used scripting language.
That's solid advice, but one has to keep in mind that (in some cases) they're locking themselves into a language.
For example, I actively try to avoid anything from Google and Facebook because I don't trust these companies.
Doesn't need to be the case, as DiTBho is showing right here. Kudos to him for biting the bullet and move to an another language!
-
Yesterday my bash libraries reached 12K lines of code.
Not for hubris talking, but this stuff is unmanageable.
So, I am going ahead for inertia, but I am already moving to Lua.
10% progress :-+
Nice ;D
-
Lua. I believe that is a purely event based script language which either originated or gained popularity as "Game engine" script languages, very popular with modders.
I've seen it used outside of that, but the brief look into it, it seemed like one of those funky languages that does everything a little different.
-
Yea. I'm curious. How do you write shell scripts in Lua?
Also, if your system needs that many/much scripting to "automate" it. It's not a very good system in my humble profressional opinion.
The analogy is giving a flight engineer a laptop to calculate the things he needs to put into the flight management systems.
Just make the flight management systems do it and fire the flight engineer entirely.
If your system has to run on the life support of so much "admin" and "maintenance" scripts, it's time to bin the system and the scripts. Rewriting the scripts in a new language is just copying your problems to the next iteration instead of fixing them. It's transcribing gibberish into the start of a new book.
Dump the baggage. Travel light.
-
Also, if your system needs that many/much scripting to "automate" it. It's not a very good system in my humble profressional opinion.
The analogy is giving a flight engineer a laptop to calculate the things he needs to put into the flight management systems.
Just make the flight management systems do it and fire the flight engineer entirely.
If your system has to run on the life support of so much "admin" and "maintenance" scripts, it's time to bin the system and the scripts. Rewriting the scripts in a new language is just copying your problems to the next iteration instead of fixing them. It's transcribing gibberish into the start of a new book.
It seems to me that the philosophy of "use scripts to customize the interactions across different independent applications" is a good one. I can't see any realistic way to change every application to do what I want, but I can easily write scripts to move this file from here to there, copy it, make an API call, or whatever else. I have a far easier time making my Linux/BSD systems "play nice" in the way that I want and find that I often simply can't do the equivalent on a Windows machine or can't do it nearly as easily.
Eliminating scripting is eliminating user control, if not in every case at least in a lot of cases. (Or perhaps I misunderstand what you're saying.)
-
Also, if your system needs that many/much scripting to "automate" it. It's not a very good system in my humble profressional opinion.
Your professional experience does not include systems integration, then.
Linux and BSD kernels are only a small part of the operating system. The majority comprises of services running in userspace. These services need to be managed, or at least started up; the kernel only initializes/executes a single userspace process as PID 1 called init (https://en.wikipedia.org/wiki/Init?useskin=vector).
To do this management, you have exactly two approaches: either you use a monolithic service supervisor, or you use a set of event-triggered scripts that are executed as the events occur. (The technologically best solutions I have seen, are a mix of the two.)
Some init systems in Linux and BSD are still based on scripts, regardless of whether they use a service supervisor or not, because it turns out to be easier to configure and maintain than Windows-type binary registry database is. Systemd uses its own format, which uses Windows-style INI files to describe the same.
Init, however, is not "all", as many applications use scripts to launch themselves (with the correct user-specific and user-locale/environment setup, typically using environment variables and command-line parameters), and many services use externally executed scripts as a way to hook into their internal events.
On my bog-standard systemd-based Linux Mint 20.3, I have 2903 executable files in /bin, /sbin, /usr/bin, and /usr/sbin. Of these, 442 (15%) are shell scripts, totalling just under 11,000 lines of shell scripts. Note that this does not include anything related to init, just launch scripts, so the total number is much, much higher on a typical system. On an embedded system with a full OS but only required applications, the number is obviously much smaller, but then, the init system is best implemented as a dedicated set of scripts optimized to minimize startup time and resource usage when the system is fully running. 12kLOC of related Bash scripts isn't excessive at all for a Linux-based embedded system or appliance, in my experience in Linux systems integration.
For further information, and to get better at systems integration, I warmly recommend the Linux from Scratch (https://linuxfromscratch.org/) project, as well as looking into the init and at least the minimum OS support services in the various BSD variants and perhaps OpenWRT (http://openwrt.org/), too. The latter is widely used in routers and switches, and most manufacturers use either a custom fork, or a similar internal builds. You'll be surprised how many scripts are used. Yet, because the scripts are only in memory during their execution, and are easily modified to adjust to different hardware and/or use scenarios, are perfect tool for their use case. An always-in-memory binary service would not only waste resources, but possibly also be slower and more fragile than independent scripts.
(Why, then, is systemd pushed so hard into Linux? The original SysV init was a good design for single-core machines without too many services. With multi-core processors, and many services and services having interdependencies, new init systems started popping up that could marshal the startups more effectively, often even in parallel, thus reducing the OS bootup startup times. For really robust service maintenance, first service supervisors for Linux appeared in late 1990s; the one I used extensively was DJB's daemontools (https://en.wikipedia.org/wiki/Daemontools?useskin=vector) and a bit later, Gerritt Pape's runit (http://smarden.org/runit/). The reason why systemd later 'won' over launchd and upstart and others, was not technical, but a business and social choice: it had the resources of Red Hat behind it. The key factor was the Debian discussion on Init systems, which caused an avalanche effect among Debian derivatives, especially Ubuntu, which had spent quite a lot of effort in developing upstart. This discussion and vote was not on technical merits, and was basically an example of how to use human resources to ensure you get the result you want; all of it is still available in Debian mailing list archives. I myself prefer technical superiority over social or business superiority, so I prefer other options. Make no mistake, systemd is nowhere near the peak of how init and service supervision can be done on current hardware; it is just the popular choice among humans.)
Would I personally want to switch from Bash to Lua? No, not really: I have so much experience in Bash that writing robust scripts in it comes naturally, by default, for me. Do I see the switch a bad idea? No, because I am interested in the technical effects of the switch. At minimum, it is an experiment I support, because I am interested in the differences to Bash/POSIX sh-based scripts. In general, I find such experiments the only real way to find a possibly better path forward; reality and practice always trumping theories.
Apologies for the wall of text. Systems integration (including scripts at the core of OS services and maintenance) is just a sensitive thing for me, because just like myself, it is often overlooked and completely ignored, while it is a key part of making an OS an useful tool; in Linux, the key difference between an unreliable and flaky OS and a robust, stable one (because typical kernel configurations tend to be absolutely rock solid). No offense or belittling intended; it is just my grumpiness due to how often systems integration gets overlooked and ignored.
-
Also, if your system needs that many/much scripting to "automate" it. It's not a very good system in my humble profressional opinion.
Your professional experience does not include systems integration, then.
Yes, or it doesn't include much work with Linux systems or similar altogether, actually.
A pretty bold and exotic statement it was.
-
Would I personally want to switch from Bash to Lua? No, not really: I have so much experience in Bash that writing robust scripts in it comes naturally, by default, for me. Do I see the switch a bad idea? No, because I am interested in the technical effects of the switch. At minimum, it is an experiment I support
I have enough experience with Bash scripting to say that beyond a certain level of complexity, such as the integration of all the various tools which are used from booting services to compiling an entire distro, Bash becomes counterproductive because it is too difficult to debug.
And that's where Lua comes in help, at least this is the experimental part I am evaluating.
but some things I'm writing in C, and I'm keeping them as binaries
/usr/bin/myfilename_adjust -> myfilename_utils
/usr/bin/myfilename_adjust_double_slash -> myfilename_utils
/usr/bin/myfilename_adjust_to_url -> myfilename_utils
/usr/bin/myfilename_deextendL -> myfilename_utils
/usr/bin/myfilename_deextendR -> myfilename_utils
/usr/bin/myfilename_delocal -> myfilename_utils
/usr/bin/myfilename_depath -> myfilename_utils
/usr/bin/myfilename_deroot -> myfilename_utils
/usr/bin/myfilename_desubfolder -> myfilename_utils
/usr/bin/myfilename_diff -> myfilename_utils
/usr/bin/myfilename_getextL -> myfilename_utils
/usr/bin/myfilename_getextR -> myfilename_utils
/usr/bin/myfilename_getleaf -> myfilename_utils
/usr/bin/myfilename_getroot -> myfilename_utils
/usr/bin/myfilename_majorchar -> myfilename_utils
/usr/bin/myfilename_upchar -> myfilename_utils
/usr/bin/myfilename_utils
/usr/mylib/lib_myfilename_utils_v9.o
/usr/mylib/lib_myfilename_utils_v9.h
/usr/mylib/lib_myfilename_utils_v9.interface
p.e. I wrote some Gentoo Overlays (recipe for how to compile and install the package) for utilities that manipulate filenames; all things that I can use in scripts, or even in other C files by linking with "lib_myfilename_utils_v9.o"
-
There are other indications that scripting (triggered by specific events, much like interrupt handlers on microcontrollers) will only become more prevalent and widely used, in time.
The two particular examples of this I know of, are various implementations of Berkeley Packet Filters (https://en.wikipedia.org/wiki/Berkeley_Packet_Filter), and Linux seccomp (https://en.wikipedia.org/wiki/Seccomp). Even Netfilter (Linux kernel firewall, used as the firewall in many routers and firewalls) can be considered one form of "scripting", although it is very specific to communications.
The Linux variant of BPF, eBPF (https://en.wikipedia.org/wiki/EBPF), is basically a simple binary scripting language that is run on the kernel side whenever triggered, for example at the (kernel-side) beginning of a syscall. It is either interpreted or JIT-compiled to native machine code; security and completion checks are done at install time. It is mostly used to construct sandboxes, so it is widely used in Android and Chromebook especially.
See the man 2 seccomp (https://man7.org/linux/man-pages/man2/seccomp.2.html) man page for usage examples in Linux.
As core counts increase, and we distribute computational tasks more and more (not necessarily to physically separate machines, but because internal buses are so short and wide that any buses between cores and bulk memory storage will still be a "bottleneck"; this is better modeled as distributed computing instead of parallel computing), I see that "scripting" using DSL's that are verified for correctness at "install"/"save" time will be needed to extract the maximum utility of such hardware.
All of these use cases have the "triggered by an event" in common. It is not worth it to keep everything in closely coupled RAM, when each event is relatively rare, and there are huge numbers of possible events. For systems integration at least, human readability is key for maintainability.
Bash [...] is too difficult to debug.
As far as I know, this is absolutely true. I am not aware of any software that would help debug Bash scripts (tethered, or as a "slave" to the debugger, like one can do with binary processes and ptrace (https://man7.org/linux/man-pages/man2/ptrace.2.html) in Linux), and while it is technically possible with GDB and suitable Python accessor helpers (a whole lot of them, too), I have always done my debugging using mark 1 eyeballs only, or by treating the Bash process as I would any binary one, tracking e.g. syscalls (strace) and/or library calls (ltrace) it does, at the machine code level.
Similarly, if you write code for restricted environments with tight security requirements, validating Bash scripts can be impossible. I definitely wouldn't want to try and do that myself.
On the other hand, because Lua implementations are embeddable with well-specified interfaces between the Lua code and the native code the interpreter runs under, it is not too difficult to develop good tools for both tasks.
Like I said, I would not do that, but I can see many reasons why someone else would, and I do support that: I do think it is worthwhile for someone to try and do this, just to find out the properties of the end result. Indeed, because of DiTBho's reply, I now think such debugging and validation tools alone might be worth it to do this as an experiment. (And if it succeeds, it gives completely new tools for systems integration, for high-reliability/high-security systems!)
-
As far as I know, this is absolutely true. I am not aware of any software that would help debug Bash scripts (tethered, or as a "slave" to the debugger, like one can do with binary processes and ptrace (https://man7.org/linux/man-pages/man2/ptrace.2.html) in Linux), and while it is technically possible with GDB and suitable Python accessor helpers (a whole lot of them, too), I have always done my debugging using mark 1 eyeballs only, or by treating the Bash process as I would any binary one, tracking e.g. syscalls (strace) and/or library calls (ltrace) it does, at the machine code level.
I find tracing (bash -x, and maybe also -e -o pipefail) very useful for debugging. You can also set $PS4 to add variables you want to watch.
-
Also, if your system needs that many/much scripting to "automate" it. It's not a very good system in my humble profressional opinion.
Your professional experience does not include systems integration, then.
Yes, or it doesn't include much work with Linux systems or similar altogether, actually.
A pretty bold and exotic statement it was.
I'm sorry, but it does include a lot of Linux work and if you go and check the history on Linux From Scratch and BeyondLinuxFromScratch I was part of the team back then and wrote a few chapters of the BLFS book. "wget" nstall and one or two others where my work.
I went further to automating the whole build from scratch to reboot.
You would be surprised at how few scripts are actually used in enterprise. They are very much limited to the very bare end of the OS.
And very, very few Unix/Linux systems use scripts for init these days. SVInit was rewritten in C decades ago and replaced with SystemD et.al. more recently.
Most enterprise systems run supervisors and hypervisors and process management systems with agents and central control.
However, the real issue is I am a software engineer and the OP I levelled my point at isn't. I get to say, "That's crap, lets write it again".
However if you are just taking about managing and maintaining an OS. That's different.
If you have two applications you own/wrote and you still need significant scripts, then you need to rewrite the application to do it's own comms and maintenance.
-
The thing that took me the most time and that, in the end, I decided to replace with C code was dependency management.
And I wrote a library that abstracts the problem and evaluates in what order something has to be planned
At the bootstrap, /sbin/init needs to know what a service needs, and calculate the minimum path for all services, in order to have the order in which to launch them, as well as it needs to know the minimum order of services that have to shut down on an event like "ops, we lost the power line and we are on battery, less than 15 min to sync disks, close services, and shut down".
On my NAS, /sbin/init is notified by a daemon that monitors the PSU over the serial line, if the power line is lost it sends a message to /sbin/init through a special pipe, and /sbin/init reacts by invoking /etc/init.d/shutdown_on_emergency, a light version of shutdown that only cares about essential services to be safely stopped without losing/corrupting data.
At the moment, C, Bash, Lua, are all involved in this process.
Something like 2 years ago, I wrote the dependency engine all in Bash, and ... in the end, I got tired of having to debug it because it took too long, so I rewrote it in C, and not only is it faster to run but it takes 1 /10 of the time to find and fix bugs.
-
As far as I know, this is absolutely true. I am not aware of any software that would help debug Bash scripts (tethered, or as a "slave" to the debugger, like one can do with binary processes and ptrace (https://man7.org/linux/man-pages/man2/ptrace.2.html) in Linux), and while it is technically possible with GDB and suitable Python accessor helpers (a whole lot of them, too), I have always done my debugging using mark 1 eyeballs only, or by treating the Bash process as I would any binary one, tracking e.g. syscalls (strace) and/or library calls (ltrace) it does, at the machine code level.
I find tracing (bash -x, and maybe also -e -o pipefail) very useful for debugging. You can also set $PS4 to add variables you want to watch.
I usually use #!/bin/bash -eufx when I don't need globbing, and #!/bin/bash -eux when I do need globbing (for some reason).
-o pipefail is ridiculous in the sense that it should be active per default IMO.
-
You would be surprised at how few scripts are actually used in enterprise.
If you limit yourself to Windows enterprise systems, perhaps.
And very, very few Unix/Linux systems use scripts for init these days. SVInit was rewritten in C decades ago and replaced with SystemD et.al. more recently.
Init is only a fraction of the total number of scripts in any Linux installation. Like I wrote, systemd replaced scripts with Windows-style INI files, and we can definitely argue whether it is an appropriate use of resources to A) parse those at runtime, or B) keep the parsed structures in memory.
You conveniently ignored my mention of the fact that on a bog-standard Linux installation 15% of executable binaries are actually launcher scripts, too. I guess facts do not exist when they go against your dogmatic beliefs?
Most enterprise systems run supervisors and hypervisors and process management systems with agents and central control.
Have you actually taken a look at how common it is for these to implement the triggers as – gah! – external scripts?
Many of these do use their own scripting language – see e.g. Puppet, Ansible, cfengine, et cetera – although I guess most "sysadmins that call themselves software engineers" today expect to use GUI interfaces to these instead of dirtying their hands with the actual implementation, much less know anything about their internal operation.
However, the real issue is I am a software engineer and the OP I levelled my point at isn't.
I am starting to seriously doubt that, actually. You make too many elementary-level incorrect assumptions and errors, and yet insist you know this stuff.
Something does not gel with the problems you discuss and your ability to solve them on your own for me to trust your skill level, especially wrt. systems integration; sorry.
I'm sure you don't trust anything I say at all either, so don't bother getting worked up: I often have the problem of people not trusting my own technical expertise. It is some sort of presentation and communications issue I have. This is exactly why I always give enough information to check the basis of my argument for themselves, because I don't even want anyone to trust me as an authority per se: I want them to check for themselves, and be critical of all expressed opinions.
To list the number of lines and the script name, I used
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 | (T=0; while read -d "" NAME ; do T=$[T+1]; file "$NAME" | grep -qe 'shell script' || continue ; wc -l "$NAME" ; done ; printf '%d files total\n' $T >&2 ) | awk ' NF>1 { L=L+$1 ; n=n+1 ; print } END { printf "%d scripts containing %d lines\n", n, L }'
The two final lines of the output contains the useful summary. On my Linux Mint 20.3,
2903 files total
442 scripts containing 109669 lines
As far as I know, this is absolutely true. I am not aware of any software that would help debug Bash scripts (tethered, or as a "slave" to the debugger, like one can do with binary processes and ptrace (https://man7.org/linux/man-pages/man2/ptrace.2.html) in Linux), and while it is technically possible with GDB and suitable Python accessor helpers (a whole lot of them, too), I have always done my debugging using mark 1 eyeballs only, or by treating the Bash process as I would any binary one, tracking e.g. syscalls (strace) and/or library calls (ltrace) it does, at the machine code level.
I find tracing (bash -x, and maybe also -e -o pipefail) very useful for debugging. You can also set $PS4 to add variables you want to watch.
I usually use #!/bin/bash -eufx when I don't need globbing, and #!/bin/bash -eux when I do need globbing (for some reason).
-o pipefail is ridiculous in the sense that it should be active per default IMO.
Making the interpreter (Bash) more verbose, does help with post-mortem and finding out exactly what went wrong, but it isn't comparable to say single-stepping or tethered debugging (as is commonly done to binaries with gdb).
There is no technical reason why a script interpreter does not provide such interfaces – single-stepping, runtime control, state examination by an external tracer process – built-in, it is just a matter of spending the time and effort for implementing and maintaining it. Thus far, nobody has seen the need for Bash (or POSIX shells, as far as I know).
Depending on which Lua interpreter implementation one wants to use, single-stepping and other runtime control may be already built-in to the implementation, and most likely one can run the same interpreter codebase with such debugging features enabled (to get a version of the interpreter one can use for debugging); and easy state examination is one reason why Lua is so easy to embed into e.g. C programs. While I haven't used Lua that much, even I can immediately see how one could make a tethered interpreter, giving a GDB-like interface for run-time debugging, single-stepping, and examination of Lua state and variables, with reasonable effort.
Thinking about this further, an init service supervisor process that uses Lua scripts with the metadata (dependencies especially) as structured comments, and event management as Lua functions (to describe when the particular service fails to start), you could actually construct a robust, verifiable init system, that would still be able to launch services in parallel.
(What we really need for service supervision, is a common library interface for acknowledging service status, for example when the service is ready to accept incoming connections, stops accepting incoming connections, has mounted a new device, has removed a device, and so on. It needs to be defined using a simple C/C++ header file, with an example dynamic library implementation that simply stubs them out, and the code to use them added to all service daemon implementations. Of course, this means that the interface has to be compatible with all, i.e. be abstract, instead of optimized towards one specific implementation. DBus low-level implementation is atrocious for this, because it requires several helper services, and is a single-point failure risk: not robust at all, so it is not suitable; but with a suitable shim library, one of the supported transports/mechanisms could be DBus.)
-
However, the real issue is I am a software engineer and the OP I levelled my point at isn't.
I am starting to seriously doubt that, actually. You make too many elementary-level incorrect assumptions and errors, and yet insist you know this stuff.
Something does not gel with the problems you discuss and your ability to solve them on your own for me to trust your skill level, especially wrt. systems integration; sorry.
I'm sure you don't trust anything I say at all either, so don't bother getting worked up: I often have the problem of people not trusting my own technical expertise. It is some sort of presentation and communications issue I have. This is exactly why I always give enough information to check the basis of my argument for themselves, because I don't even want anyone to trust me as an authority per se: I want them to check for themselves, and be critical of all expressed opinions.
I think like most cases of technical disagreement we are talking from and about different perspectives.
You sound to me like a sys-admin and a solution integrator. Nothing wrong with that but it's not really software engineering. Partly, there will need to be some project engineering if it can't be done by a single person.
This is a common and repeated problem in "technical suburbs". Just like I an a complete dweeb about electronics, most people on here think that writing code and scripts is software engineering.
People were writing code and scripts decades before we had software engineering. Software Engineering is what happened in response to the 1960s, 1970s and beyond which was caused by spaghetti scripts, code and "single man band" projects which nobody else understood. Literally billions of pounds is being spent STILL to this day fixing that 'ing mess.
Having 3 small applications and connecting them up with pipes, sockets etc. is fine at a low operating system level.
I haven't written an application which produced anything on standard out or standard in, professional in over 10 years.
The only shell script I have written in the past year was a very carefully constructed admin "purge" job which was ONLY commissioned because the application had been hurried/forced into release without it's own admin.
-
Inversely, for development, I am lucky enough to have changed jobs to where I can at least run more development systems locally. As such the first thing I installed was GitBash. Then WSL. Then Docker.
The target platform is linux, but I do not foresee a large amount of scripts. Certainly not bash scripts. Nothing exists as files for a start, there is virtual no "set down state" and the only "in memory" state is held in middle ware. Other than starting/stopping I don't see the need for many scripts.
Remember we were talking about a single user being responsible for 12K lines of bash code. That is what brought me to suggest something is wrong in that overall system.
I have written hobby systems in Bash alone. Perl as well. They are great for "admin" and "hacks" and OS integration or portability, but invariably the systems become combersome to deal with in shell and migrate to something like Python instead.
-
As an example of a day job in software engineering.
I am prototyping and demoing service granularity in both configuration layer and the functional layer.
Problem: Big fat monolithical services/applications become a bear to maintain and extend.
Solution: Fragment them into many little services and have them collaborate via middleware (message bus etc.)
Problem: Having a few dozen git repos and a few dozen Jenkins build artifacts, several dozen docker images and a whole end to end maintenance, governance, security, version matrixes, release strategies and testing matrixes, etc, etc, for each... is a pain in the backside.
Solution: Mono or Uber repos. Place more than one service per repo. In fact bundle all related (by data or by tech/dependencies /integrations) into a single repo.
One step further: Polymorphic services. You would be familiar with this as the "symlink busybox pattern" in linux boot discs. Basically it's one application, but when you launch it, you selective enable components. Thus you can deploy all dozen services in one go to one "pod" or you can deploy a spread of them to meet the load of each step individually in an elastic fashion.
As to the code that goes into these services. That's the easy part.
-
You sound to me like a sys-admin and a solution integrator.
It is only a fraction of my background, the one that happens to be relevant here. I have also been paid to write code in at least half a dozen programming languages.
Note that this is the Programming sub-subforum of the Computers subforum; not the Embedded one. We're not talking about single-binary appliances, but systems with a proper kernel-userspace boundary, i.e. fully hosted systems in the C/C++ standards definition.
I already explained why 12k lines of Bash in a fully hosted Linux system, even in an embedded appliance, is nothing strange. I also showed how even a basic installation of desktop Linux Mint 20.3 using systemd init yields an order of magnitude more lines, ~110k, in Bash or POSIX shell launcher scripts in /bin, /usr/bin, /sbin, and /usr/sbin, and how to check the same on any Linux, BSD, OpenWRT, etc. system.
The fact that you still object, without any basis other than your own assertions, is the indicator that you really don't know what you are talking about.
I'm sure that kind of marketing tactic will get you far in any business organization, as they're only concerned with short-term and surface appearances, but here, among other technically adept people, you do have to base your boasts on verifiable facts, not your own authority or CV.
The only shell script I have written in the past year was a very carefully constructed admin "purge" job which was ONLY commissioned because the application had been hurried/forced into release without it's own admin.
There are software designers who believe they know how the software they designed should be allowed to be used, and then there are those who design tools to be used in whatever way users find appropriate, verified to accomplish those tasks in at least one way, but not limited to that one.
I've met a metric fuckton of Windows-based developers who believe only the former matters. They consider the Unix principle ridiculous and outdated, even though it is what still drives basically all internet-facing servers, except for a small handful of Windows machines. Perhaps they write quite sensible code for Windows, but I've yet to find one that can also design acceptable applications and services for other OSes, especially POSIXy ones. It is like their mind has been reprogrammed, or something.
Interface modularity is something you get if you understand how and why one would use scripts instead of libraries. The former is easily modified to adjust to each task, including new interfaces that appear after the application or service is in production, and the latter is when you provide a set of solutions to the end users, updated only if the entire application/service/system is upgraded (i.e., never). It is at the heart of the Unix philosophy, although the aforementioned Windows types do seem to object to it too, ignoring a few decades of successful implementations.
Interface modularity is exactly why I prefer to write graphical user interfaces in Python 3 and Qt 5, with the high-bandwidth, possibly properietary Secret Sauce written in C (for better portability, as C++ requires version-specific libraries) and dynamically linked to the Python process. Documenting the interface (fully for the Python side) is sufficient boundary for using GPL-licensed Qt, and any open source license (even a proprietary, non-copyleft open source license) for the Python interface, and any (open source or not, proprietary or copyleft) license for the C code.
Why so? Because the user interface part is the one users have most issues with. By designing the entire application so that it allows end users to modify (at their own risk) the user interface, without giving them access to the actually meaningful/sensitive/protection-worthy C code, or without them having to install any kind of development environment, you maximize the end user freedom with minimal business risk.
A big reason for the popularity of Inkscape is its extensions (https://inkscape.org/develop/extensions/) system, which allowed users and developers to write import and export filters for a surprising number of vector graphics formats. It, too, supports external "scripts" (any executable will do), with an XML description of the extension UI defining the user choices and passing those as command-line parameters to the external executable. The data extensions work on, is always in SVG format. Inkscape even provides inkex.py, a Python module that makes writing Inkscape extensions in Python quite easy.
With the entire user interface being user-modifiable, users willing to take the risk can fix and adjust the UI to best fit their workflow. With only a small marketing push, a company can do e.g. yearly competition on best UI modifications, and provide a repository of such unofficial and untested interfaces submitted by other users.
Of course, if your "software engineers" believe only monolithic and fully proprietary applications are the only sensible way to construct applications, they will never accept that or say using user-specified external programs or scripts to work on that user's data. After all, all data belongs to the application, not the user, eh?
-
I am also not interested in any kind of "winning" in this discussion, either. What I'd really want to see, is paulca checking all the files provided in their most recent full-Linux (kernel and userspace) appliance, and counting the number of shell scripts and the number of lines in them, and reporting it here.
Either the figure is basically zero, in which case they have a point, or it isn't and they do not. Facts are facts, and I want to make them visible over all the opinion-based fluff.
When you provide fully integrated appliances, say an OS controlling some industrial device, you are responsible for the entire product. Just claiming that a bug was in an upstream-provided script is not a valid answer. The 12kLOC of Bash DiTBho mentioned is pretty darn close to a minimum used in a stripped-down but functional appliance-style Linux installation. DiTBho can clarify, but I assumed that is the number of lines of shell scripts in the full system, not just in some custom init system or such; i.e. includes all the scripted parts that one needs to replace or verify, if they want to be able to answer for the entire system without resorting to the excuse at the beginning of this paragraph.
For a realistic real world data point on the use of scripts, perhaps looking at OpenWRT would be in order? It is widely used, used on very limited hardware (currently 8 MB of storage, 64 MB of RAM), and a very good example of a widely-used appliance. (The squashfs for openwrt-22.03.5-ramips-mt7620-asus_rt-ac51u-squashfs-sysupgrade.bin has 222 executable shell scripts, totaling 8514 lines; or 232 scripts that file reports as something 'shell script' totaling 8943 lines. Only about two thirds of the mentioned 12kLines, sure, but AFAIK DiTBho works on quite different types of appliances, and this is only a very basic router installation.)
-
To list the number of lines and the script name, I used
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 | (T=0; while read -d "" NAME ; do T=$[T+1]; file "$NAME" | grep -qe 'shell script' || continue ; wc -l "$NAME" ; done ; printf '%d files total\n' $T >&2 ) | awk ' NF>1 { L=L+$1 ; n=n+1 ; print } END { printf "%d scripts containing %d lines\n", n, L }'
And that's the proof that bash sucks for anything than basic stuff. :-DD
-
Very concise, there's no denying that.
But it takes a lot to unravel it. Pipes, wc, find, awk, grep ... all together.
I appreciate that a single line program is capable of doing all that, but there have to be simpler ways to do the task, even if it entails writing more lines of code.
-
To list the number of lines and the script name, I used
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 | (T=0; while read -d "" NAME ; do T=$[T+1]; file "$NAME" | grep -qe 'shell script' || continue ; wc -l "$NAME" ; done ; printf '%d files total\n' $T >&2 ) | awk ' NF>1 { L=L+$1 ; n=n+1 ; print } END { printf "%d scripts containing %d lines\n", n, L }'
And that's the proof that bash sucks for anything than basic stuff. :-DD
Well, it does use find, bash, file, grep, wc, and awk to get the work done.
You need to first construct a list of executable files. In Linux, all bytes except NUL and slash are valid in names, so I used -print0 to print the path to each file NUL-separated.
Next, you need to pick the files that are shell scripts, just the task for file and grep -qe 'shell script', as the former will include 'shell script' for POSIX ('POSIX shell script') and Bash ('Bourne-Again shell script'), and the latter will return success iff found, failure otherwise. I used a Bash/POSIX shell loop, with continue skipping all non-matching file types. For shell scripts, we want the line count, which wc was designed to report. The output is one line per file, with the line count first and then the file name or path. The loop also counts the total number of files seen, and prints the total count to standard error. For this to work, the entire loop section must be in a subshell.
Finally, the awk part counts the number of shell script files, and sums up the total number of files, printing each input record as it goes.
Because any file names with newlines will make the wc part confuse awk, it does not actually work for all possible file names, though.
This is better:
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 | xargs -r0 file -00 | gawk 'BEGIN { RS=FS="\0" } { files++; name=$0; getline; if ($0 !~ /shell script/) next; scripts++; n=0; RS="\n"; while (getline < name) n++; close(name); RS="\0"; lines+=n; printf "%9d %s\n", n, name } END { printf "%d files, of which %d shell scripts, containing %d lines\n", files, scripts, lines }'
or split into logical lines,
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 \
| xargs -r0 file -00 \
| gawk '
BEGIN {
RS=FS="\0"
}
{
files++
name=$0
getline
if ($0 !~ /shell script/) next
scripts++
n=0
RS="\n"
while (getline < name) n++
close(name)
RS="\0"
lines+=n
printf "%9d %s\n", n, name
}
END {
printf "%d files, of which %d shell scripts, containing %d lines\n", files, scripts, lines
}'
The "find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 | xargs -r0 file -00" part of the command produces a sequence of "filename\0type\0", i.e. filename and type as text, separated by NULs. It does this by piping the NUL-separated file names or paths (-print0), to xargs, which splits them back (-0 NUL-separated, and -r says not to run if there are no parameters to supply), and executes file -00 with each file name or path as a separate parameter, as many as can fit at a time. The -00 parameter tells file to output a single NUL after each filename and after the type of the preceding file.
Other than executing and piping the commands together, it does not use Bash at all. gawk can handle NUL-separated records by simply setting RS="\0". I set also the field separator to the same, because we don't want gawk to waste time splitting records (or lines) into fields. The main rule gets applied to each executable file found in those directories. If file did not report it as a shell script, it is only counted (as a file, not as a script). For shell scripts, we count the number of lines by reading each line (using newline as the record separator), and print the per-script count, and update the tally. The END rule is applied after all input has been processed, and it prints the summary.
In POSIX C, we could use nftw() to walk the trees, or scandir() to obtain the list of files in each directory. After checking the stats of the file (to make sure we only look at group-executable files), we can read the first line of each file using getline(), and if it looks like a valid shebang line for bash, dash, ash, or sh, count the number of lines in it (by reading each line), and otherwise ignore it. Count the number of files, the number of scripts, and the number of lines in script files, and print them, and you're done.
Do note that while the shell script stanza is concise, it also uses more than one process at a time. xargs will buffer file names so that we execute file the fewest number of times; it is faster than executing it once for each file (which you can do by using xargs -r0 -n 1 file -00 instead). The piped processes are run in parallel, and they are executed all at the same time, which means that on multi-core machines you use at least three cores if available. To do the same in C, you'll need to use threads, since otherwise a single core (at a time) will be used to perform the tasks. (You can use e.g. popen(), but executing anything once for each file found will be slow.)
-
I started looking for a way to program the same application in Python and I have found that it is necessary to replicate the operation of standard unix tools like "file", "find", "wc", "grep", etc.
Some, like grep or wc, are easy to implement. Others like find are not so easy or immediate. Finally "file" needs an external library.
For some reason Unix tools have been around for so long in "good health".
-
functions are not actually functions
-> can override without a warning
-> no actual return value
-> the number of arguments is not checked
-> arguments data-type is not checked
the integer data type is simulated by external tools
-> compare two numbers is ugly, and too error prone
Those may be arguments not to want to write shell scripts, but not to rewrite scripts that will probably have been tested for years or even decades on many systems.
I believe most modern systems don't actually use bash for init scripts, but something lighter weight like ash. Sure, the language is ugly, but there's a reason why it has been used for simple scripts for decades.
Perl and Ruby would be the obvious candidates in my mind, but I'm not so sure if they are much lighter than Python. If you want very low footprint, try Forth. It's used in some bootloaders.
I'm surprised Forth has not been discussed more. I know some people just can't accept a language that isn't actually compiled or that uses reverse polish notation. Sometimes it's surprising the limitations people put on themselves. Forth is an amazing tool for all sorts of work, especially close to the metal. It's what I use in my FPGA designs when I want a processor.
-
I started looking for a way to program the same application in Python and I have found that it is necessary to replicate the operation of standard unix tools like "file", "find", "wc", "grep", etc.
Uh? These are all individual programs. You can call them from Bash, but otherwise from any other scripting language that allows executing external programs, which both Lua and Python can.
Bash doesn't do anything else than spawning processes for any command you ask it to execute and that's something you can do in many languages.
The only thing that may be more or less difficult in other languages is directly piping one executable to another - that would be something to check.
For Python, this is where to start: https://docs.python.org/3/library/subprocess.html
Some, like grep or wc, are easy to implement. Others like find are not so easy or immediate.
Again why would you necessary want to reimplement them? Unless you're not talking about the topic, which was to replace Bash with another language, but talking about something different, like being able to run the same script on environments which do not provide the above listed tools, which would be a completely different endeavor. Bash and the standard command-line tools commonly found on Unix-like systems are two completely different things.
-
To list the number of lines and the script name, I used
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 | (T=0; while read -d "" NAME ; do T=$[T+1]; file "$NAME" | grep -qe 'shell script' || continue ; wc -l "$NAME" ; done ; printf '%d files total\n' $T >&2 ) | awk ' NF>1 { L=L+$1 ; n=n+1 ; print } END { printf "%d scripts containing %d lines\n", n, L }'
And that's the proof that bash sucks for anything than basic stuff. :-DD
Here are the parts of it that are bash:
… | (T=0; while read -d "" NAME ; do T=$[T+1]; … | … || continue ; … ; done ; printf '%d files total\n' $T >&2 ) | …
In another language each "…" is typically written as system() or popen() containing exactly the same as whatever I replaced with the "…".
-
In another language each "…" is typically written as system() or popen() containing exactly the same as whatever I replaced with the "…".
Probably not. In higher level languages you rarely resort to using the command line interface. |O
Here are the parts of it that are bash:
… | (T=0; while read -d "" NAME ; do T=$[T+1]; … | … || continue ; … ; done ; printf '%d files total\n' $T >&2 ) | …
Doesn't matter, the result is the same: an unreadable mess.
-
I can read and parse (and even debug and maintain) such stanzas easily, but that's because some of my work involved a lot of conversions, generation and/or slicing of text-based data sets (variants of PDB format data (https://en.wikipedia.org/wiki/Protein_Data_Bank_(file_format)) for molecular dynamics simulations); my heavy use of awk is also from that era.
(But give me a few lines of simple Perl, and I break out in hives, due to a horrible experience maintaining some Perl code ages ago.)
When using syntax highlighting, the fixed stanza becomes much more readable:
(https://www.eevblog.com/forum/programming/what-would-you-use-to-replace-bash-for-scripting/?action=dlattach;attach=1811131;image)
(Apologies, I'm too lazy to convert the syntax highlighting to forum text colors; an image has to suffice.)
In POSIX C, one can use ssize_t len = getdelim(&line, &size, '\0', stdin); (with previously used and/or originally initialized char *line = NULL; size_t size = 0;) to read NUL-delimited inputs. It is an excellent way to receive e.g. paths from an external script (replacing stdin with a handle returned by popen(script_path,"r")) that does file format detection or other user-specified custom filtering, while still supporting all possible file names and paths in Linux. If you have more than one field, like file -00 provides pairs of NUL-terminated string, use different line pointer and size for each field. After all input is done, len is negative, and one can safely free() all line pointers (it is safe to do even if they are NULL).
I warmly recommend making sure you have similar abilities in Lua. File names and paths in Linux really do need to be treated as arbitrary byte sequences with path elements separated by '/' (47) and terminated with NUL (0). For even the above stanza to work, one may need to set LANG=C LC_ALL=C, if some file names contain illegal byte sequences for the current locale charset; this is typical if you have a partition with Windows 1252 -encoded file names mounted, but current locale uses UTF-8 character set.
The main use of shell scripts is to efficiently chain and combine simple utilities. This is worth it, because of combinatorial explosion (https://en.wikipedia.org/wiki/Combinatorial_explosion): the number of possible combinations is much more than just the sum of options in each utility.
Consider, for example, how many different ways shells makes pipe use easy: You have $(...) (and equivalent old `...`) to insert the output of an executable (or subshell); ...|... to construct a pipe from the output of the left side to the input of the right side; and Bash <(...) which expands to the path of the read end of a pipe with its write end connected to the output of the executable or subshell. Both sides of each pipe execute concurrently, too, making easy use of multiple CPU cores if available.
A portable launcher script written in Bash or POSIX shell might contain
case "$(uname -o)" in
to set command-line arguments or environment variables based on the OS, and "$(uname -r)" if there are libraries or options that should be enabled only for specific kernel versions. It might even create a temporary directory (using mktemp -d), and populate it with symlinks to all libraries needed by the application at runtime, and point LD_LIBRARY_PATH environment variable to that, so that at each invocation, the compatible system-installed libraries will be used, and the rest used from a non-system application-specific directory suitable for that kernel.
-
In another language each "…" is typically written as system() or popen() containing exactly the same as whatever I replaced with the "…".
Probably not. In higher level languages you rarely resort to using the command line interface. |O
In other words, you simply don't want to do "scripting" [1] at all, whether in bash or otherwise.
Which is fine, but it makes your scripting opinions irrelevant in this thread.
[1] programatically combining the functions of a number of specialised stand-alone programs to achieve a desired result.
-
Statistically, it can be interesting to look at the distribution of the number of lines in scripts (including the first, 'shebang', line; i.e. #!/bin/bash or equivalent, and including empty and comment lines). For example,
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 \
| xargs -r0 file -00 \
| gawk 'BEGIN {
RS=FS="\0"
}
{
name=$0
getline
if ($0 !~ /shell script/) next
scripts++
n=0
RS="\n"
while (getline < name) n++
close(name)
RS="\0"
lines[n]++
if (maxlines<n) maxlines=n
}
END {
count=0
for (n=0; n<=maxlines; n++)
if (lines[n]>0) {
count+=lines[n]
printf "%.1f%% have at most %d lines\n", 100*count/scripts, n
}
}'
on my system, half of the launcher scripts have 75 lines or less, and a quarter are just 14 lines or less.
To skip empty and comment lines, including the initial shebang line, we can use a somewhat slower
find /usr/bin /usr/sbin /bin /sbin -maxdepth 1 -type f -perm /o+x -print0 \
| xargs -r0 file -00 \
| gawk 'BEGIN {
RS=FS="\0"
}
{
name=$0
getline
if ($0 !~ /shell script/) next
printf "%s\0", name
}' \
| while read -d "" name ; do \
sed -e '/^[\t ]*#/d' -e '/^[\t ]*$/d' "$name" | wc -l
done \
| awk '{
scripts++
lines[$1]++
if (maxlines < $1) maxlines=$1
}
END {
count=0
for (n=0; n<=maxlines; n++)
if (lines[n]>0) {
count+=lines[n]
printf "%.1f%% of scripts (%d) have at most %d lines\n", 100*count/scripts, count, n
}
}'
which drops the median count to 51 lines of code, and a third having only 17 lines of code or less, and two thirds having only a hundred lines or less.
The Bash while loop in the middle is useful, because we can then use sed to skip lines we are not interested in (those beginning with # indicating a comment, or empty), and just calculate the number of lines in the result. At this point, we're no longer interested in the file names, so the output of the loop is only the number of non-empty, non-comment lines in each shell script executable.
This means that it is reasonable to expect typical scripts to be quite short.
If we expand the search (find /usr /bin /lib /etc -type f -perm /o+x -print0), I find 3090 executable shell scripts on my system under these directory trees, with 72.7% at most a hundred LOC, half at most 43 lines, and one fifth ten lines or less, not counting shebang or comment or empty lines; further supporting that argument (since this is a deliberately typical Linux Mint installation I haven't modified).
What does it mean for scripts to be short? In my opinion, it supports my argument (https://www.eevblog.com/forum/programming/what-would-you-use-to-replace-bash-for-scripting/msg4920271/#msg4920271) that scripts are extremely useful as modular interface glue.
_ _ _ _ _
When running various simulations and collecting data, I do not include any visualization libraries to the simulator; I output data in a hopefully efficient form that I can later (or concurrently during simulation) feed to various different visualizers – even on completely different machines.
For verifying the structures of trees, heaps, and forests, I often emit the graphs in Graphviz DOT format; output statistical data in plain decimal for graphing in gnuplot, and so on.
The downside (if you consider that), is that you have a lot of intermediate data, and its temporary storage does require pretty good bandwidth to keep up. (This leads to things like using M.2 PCIe SSD's in Linux in sw-RAID0/1 configurations, and other similar shenanigans.)
The upside is that the simulation process is clean of any visualization, reducing unnecessary complexity, making maintenance easier, and that I can easily control the CPU and I/O priority of the simulation(s) separately from the conversions, which is quite useful with limited computing resources.
_ _ _ _
There is always a tradeoff in selecting ones tools. I do not think scripts are any kind of panacea; their application is always some kind of a tradeoff.
The walls of text I've posted to this thread is not intended to get anyone to use scripts more, but to show possible cases where they are worth their tradeoffs (that do depend on the language and interpreter used!).
Rejecting all script use wholesale is rejecting an entire class of tools. Like saying nails are unnecessary because screws are better. I am not suggesting using nails instead of screws to build houses and such, just pointing out that things like finishing nails and pin nails are actually ubiquitous, in furniture, molding, etc; they're just often hidden, so you don't notice them unless you look/check.
What DiTBho is doing by replacing Bash with Lua is in my opinion like choosing a different material (than the standard galvanized or nickel-plated steel) for such finishing nails and pin nails, because of their better chemistry and longevity in the surrounding material, and possible new ways of examining their reliability and efficacy. It is a lot of work, but there is also potential there. Only the results will tell for sure.
-
In another language each "…" is typically written as system() or popen() containing exactly the same as whatever I replaced with the "…".
Probably not. In higher level languages you rarely resort to using the command line interface. |O
In other words, you simply don't want to do "scripting" [1] at all, whether in bash or otherwise.
Which is fine, but it makes your scripting opinions irrelevant in this thread.
[1] programatically combining the functions of a number of specialised stand-alone programs to achieve a desired result.
I don't know what you are talking about.
Your definition of "scripting" seems to be very odd.
Wikipedia both lists PHP and JavaScript as "scripting" languages, where you usually don't interact with other programs (awk, sed etc.) directly.
-
I started looking for a way to program the same application in Python and I have found that it is necessary to replicate the operation of standard unix tools like "file", "find", "wc", "grep", etc.
Uh? These are all individual programs. You can call them from Bash, but otherwise from any other scripting language that allows executing external programs, which both Lua and Python can.
Bash doesn't do anything else than spawning processes for any command you ask it to execute and that's something you can do in many languages.
The only thing that may be more or less difficult in other languages is directly piping one executable to another - that would be something to check.
For Python, this is where to start: https://docs.python.org/3/library/subprocess.html
I am familiar with the subprocess lib. I have used it frequently to call some external tools like ImageMagick that sometimes seem to me better than the corresponding Python library (PIL / Pillow).
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
Some, like grep or wc, are easy to implement. Others like find are not so easy or immediate.
Again why would you necessary want to reimplement them? Unless you're not talking about the topic, which was to replace Bash with another language, but talking about something different, like being able to run the same script on environments which do not provide the above listed tools, which would be a completely different endeavor. Bash and the standard command-line tools commonly found on Unix-like systems are two completely different things.
I have no complex in declaring myself a Windows user. I'm sure there are many on the forum who look down on this option, but it seems to me an artificial posh position.
What I was getting at. In Windows I use Cygwin intensively and I could hardly do without the Unix environment it provides (especially bash and make). I have also used sometimes the linux environment for Windows WSL.
So I don't understand what you mean when you talk about environments that don't provide the standard Unix tools.
-
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
"Messy" and "harder" are opinions. I've found mine varies as time goes on.
"Slower" is complicated.
Executing an external process, be that a script or whatever, always has some latency.
External processes can run concurrently, on different cores, than the main process.
Data transfer between the parent and external process takes extra time, because the actual data needs to be copied (due to process separation).
Thus:- We can safely say that an external script or program does require more CPU effort (at minimum, the new process setup, and transferring the data between processes).
- We can not safely say that using an external script or program will always take more wall-clock time, especially if the main process is single-threaded, or CPU-limited (heavy computation).
- We cannot say that using less CPU time makes a process feel faster to us humans.
One "trick" I often use with my own MD simulators, where I want to save a snapshot of the system state every N time steps, is to simply copy the state to a pre-reserved area in memory, and have a dedicated thread (or MPI async send) write/send the data to storage, with the simulator only "paused" for the duration of the in-memory copy, not the write to the storage. The simulation has often proceeded several time steps, before all of the snapshot data has been saved. If I had done the save/send synchronously, the simulation would have been that much slower per wall clock time elapsed.
A practical example: Consider when you have an application where the Save operation takes a while.
If you do it synchronously, the user cannot proceed before the operation completes.
If you make an in-memory snapshot, and then delegate the saving of that snapshot to a separate thread, you can let the user do whatever they want, including modifying the data, and simply show a progress meter indicating how the save is proceeding. If the user tries to close the program, you do need to make them choose between waiting or cancelling that previously started operation, so it does add a bit of complexity.
Most application developers do not want to implement such in-memory snapshots, in particular because the saved state tends to be scattered in memory, and making an in-memory copy of it just to format and save it to a file is messy and hard. Typical solutions to the "don't freeze the interface while saving" lead to an implementation where the saved data may be inconsistent, if the user modifies the data to be saved before the save completes. (WontFix: Wait for the save to complete before you modify the document, user.)
When you implement the in-memory snapshot for file saving, say in Python and Qt for example, you'll find that the machinery you constructed for that, is just about perfectly suited to passing that data to and from helper processes. So, there are many user-facing applications that can be made much nicer ("faster", more responsive) for us humans to use, if one applies the techniques that one would use to pass the data to external programs, even if one does not end up using external programs or scripts. It just has to be designed in properly, and not tacked on as the only solution that lead the spaghetti to stick to the wall.
-
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
"Messy" and "harder" are opinions. I've found mine varies as time goes on.
I have to agree with Picuino here, it's not an opinion, it's fact.
-
I started looking for a way to program the same application in Python and I have found that it is necessary to replicate the operation of standard unix tools like "file", "find", "wc", "grep", etc....
I finally got around to writing an involved script-style application of the sort I used to write with awk, expr, sed, sh and friends, but this time I used Python. At least for me, I have found it easier to just go ahead and replicate the bits of the standard UNIX tools I need and build up the scripting to a much higher level – as in, address the details and then forget about them. As an example, I need to scan the files in a directory, from oldest to newest, so that I process the few that have become stale and skip everything newer. What I ended up with was faster and easier, in development and execution (the latter out of Python), than using the standard UNIX tools.
@classmethod
def get_wip(cls):
"""Return the download times of the emails that are Work In Progress.
Return a dictionary of the download time, in seconds since the
epoch, for each header hash / download-time file name in the WIP
directory.
"""
# Don't read the files just yet (see identifiers()). Using a Unix shell
# command (e.g., ls -tr) to list the oldest to newest header-hash file
# names from a WIP directory containing 1000 or 10000 files is wordier
# and slower than the following dictionary comprehension.
#
times_dir = root_dir/'WIP'/'' + os.sep # Create fast plain str
return {cls(name): os.path.getmtime(times_dir + name)
for name in os.listdir(times_dir)}
-
In another language each "…" is typically written as system() or popen() containing exactly the same as whatever I replaced with the "…".
Probably not. In higher level languages you rarely resort to using the command line interface. |O
In other words, you simply don't want to do "scripting" [1] at all, whether in bash or otherwise.
Which is fine, but it makes your scripting opinions irrelevant in this thread.
[1] programatically combining the functions of a number of specialised stand-alone programs to achieve a desired result.
I don't know what you are talking about.
Your definition of "scripting" seems to be very odd.
Wikipedia both lists PHP and JavaScript as "scripting" languages, where you usually don't interact with other programs (awk, sed etc.) directly.
Oh, really?
var shell = require('shelljs');
if (!shell.which('git')) {
shell.echo('Sorry, this script requires git');
shell.exit(1);
}
// Copy files to release dir
shell.rm('-rf', 'out/Release');
shell.cp('-R', 'stuff/', 'out/Release');
// Replace macros in each .js file
shell.cd('lib');
shell.ls('*.js').forEach(function (file) {
shell.sed('-i', 'BUILD_VERSION', 'v0.1.2', file);
shell.sed('-i', /^.*REMOVE_THIS_LINE.*$/, '', file);
shell.sed('-i', /.*REPLACE_LINE_WITH_MACRO.*\n/, shell.cat('macro.js'), file);
});
shell.cd('..');
// Run external tool synchronously
if (shell.exec('git commit -am "Auto-commit"').code !== 0) {
shell.echo('Error: Git commit failed');
shell.exit(1);
}
-
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
"Messy" and "harder" are opinions. I've found mine varies as time goes on.
I have to agree with Picuino here, it's not an opinion, it's fact.
Asserting an opinion does not a fact make.
-
Oh, really?
var shell = require('shelljs');
if (!shell.which('git')) {
shell.echo('Sorry, this script requires git');
shell.exit(1);
}
// Copy files to release dir
shell.rm('-rf', 'out/Release');
shell.cp('-R', 'stuff/', 'out/Release');
// Replace macros in each .js file
shell.cd('lib');
shell.ls('*.js').forEach(function (file) {
shell.sed('-i', 'BUILD_VERSION', 'v0.1.2', file);
shell.sed('-i', /^.*REMOVE_THIS_LINE.*$/, '', file);
shell.sed('-i', /.*REPLACE_LINE_WITH_MACRO.*\n/, shell.cat('macro.js'), file);
});
shell.cd('..');
// Run external tool synchronously
if (shell.exec('git commit -am "Auto-commit"').code !== 0) {
shell.echo('Error: Git commit failed');
shell.exit(1);
}
Yes, really. shelljs does not use the shell, it has a misleading library name and you fell for it. :-DD
shelljs is just a wrapper around native JavaScript APIs like the fs module etc.
For example, shell.sed does indeed NOT call the sed binary but instead emulates the expected behaviour with native JavaScript.
You can see this for yourself here (https://github.com/shelljs/shelljs/blob/master/src/sed.js#L79):
var result = lines.map(function (line) {
return line.replace(regex, replacement);
}).join('\n');
The same can be said about shell.ls, it doesn't use the ls binary at all but emulates the behaviour of ls in pure JavaScript (see here (https://github.com/shelljs/shelljs/blob/master/src/ls.js)).
As it should be. It would be idiotic to use the shell in a higher level language where you have the APIs available.
shelljs is just a library that emulates commonly used shell commands like cp, rm etc. but doesn't use them AT ALL.
What you did is proving my point, nobody uses the shell / external commands in JavaScript if they don't have to.
This is because of the following reasons:
- Command line tools are designed to be used by humans, not other programs (generally speaking)
- Command line tools, although there's some standardization with POSIX, are generally not-portable
- APIs can (and are) be versioned, command-line tools not so much.
- Because of API versioning, APIs tend to be very stable
- APIs (at least, native language APIs) are cross-platform and have the same behaviour across platforms
- ... and some more
shelljs appears to cater to people accustomed to the UNIX command line - but that doesn't change the fact that this library isn't using the shell / external programs at all (which was my initial point).
-
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
"Messy" and "harder" are opinions. I've found mine varies as time goes on.
I have to agree with Picuino here, it's not an opinion, it's fact.
Asserting an opinion does not a fact make.
It's not an opinion but general knowledge. :-// :-DD
-
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
"Messy" and "harder" are opinions. I've found mine varies as time goes on.
I have to agree with Picuino here, it's not an opinion, it's fact.
Asserting an opinion does not a fact make.
It's not an opinion but general knowledge. :-// :-DD
Just like Earth being flat, and the Sun orbiting the Earth. Right. :palm:
-
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
"Messy" and "harder" are opinions. I've found mine varies as time goes on.
I have to agree with Picuino here, it's not an opinion, it's fact.
Asserting an opinion does not a fact make.
It's not an opinion but general knowledge. :-// :-DD
Just like Earth being flat, and the Sun orbiting the Earth. Right. :palm:
I beg your pardon, that's a pretty ridiculous comparison.
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
Is true. Just as the earth is round whether you believe it or not :-DD :palm:
-
While I get your point and I personally do not like restoring to `system`, `popen`, etc. Sometimes it's well worth doing so even if it's possible to achieve what you want in your own application using native library calls, etc.
The plethora of applications that invoke `ffmpeg` or `vlc` are perfect examples of this, implementing it directly is tricky and time consuming. What about programs that check on your system status for monitoring like Zabbix/Nagios, etc. These all invoke command line tools to determine the status of applications. Tools like Puppet & Saltstack for system configuration and deployment extensively make use of shell commands.
Or, lets turn to PHP. Until gd2 became common place everyone used `imagemagik` via system commands to manipulate images. Want to recursively chown a directory in PHP with specific access permissions? sure you can write a function to do that, or just use `exec('chown -R u=rw,g=rw,o=r,a+X ' . escapshellarg($path), $out, $errcode);` Which is easier/faster and considered more reliable?
Just because you can re-implement something in your application to do it directly doesn't always make it the right or wisest choice.
-
While I get your point and I personally do not like restoring to `system`, `popen`, etc. Sometimes it's well worth doing so even if it's possible to achieve what you want in your own application using native library calls, etc.
The plethora of applications that invoke `ffmpeg` or `vlc` are perfect examples of this, implementing it directly is tricky and time consuming.
That's why I said:
... , where you usually don't interact with other programs (awk, sed etc.) directly. ...
... nobody uses the shell / external commands in JavaScript if they don't have to. ...
I'm not sure if ffmpeg has native API bindings in nodejs, PHP etc. but in the case of ffmpeg the case is pretty clear: you don't want to reimplement it.
In other cases, which have been provided by brucehoult the case is clear: you don't want to invoke sed if you can use the native RegEx engine. It's just ridiculous. :palm:
Just because you can re-implement something in your application to do it directly doesn't always make it the right or wisest choice.
No, but using the shell for basic operations like "delete this directory" or "copy this directory" is pretty lame :-DD :-DD and will make you look like a joke to experienced high level programmers ;D ;D
You should default to using native APIs, then resort to external commands / the shell. Not the other way around :palm: :palm:
All in all, brucehoult's lack of experience in higher level language is clearly showing, which is fine.
Which is fine, but it makes your scripting opinions irrelevant in this thread.
Just be a little more careful when dismissing other people's opinion :-DD
:-/O
-
In other cases, which have been provided by brucehoult the case is clear: you don't want to invoke sed if you can use the native RegEx engine. It's just ridiculous. :palm:
Not really, it depends on what you're doing. Try using your native RegEx engine on a huge file for example, you now suddenly need to deal with reading the file as a stream as loading the entire file into RAM would be a huge waste of resources. The complexity of what you're implementing suddenly goes up and it may not be worth the time/effort or cost to implement all the edge case error handing, etc.
Just because you can re-implement something in your application to do it directly doesn't always make it the right or wisest choice.
No, but using the shell for basic operations like "delete this directory" or "copy this directory" is pretty lame :-DD :-DD
Not when you're dealing with a huge directory structure and you're coming from a scripted language where it's infinitely faster to use `rm` or `cp`. Again it depends on your use case, you might even want to take advantage of `rsync` to avoid unnecessary copying of files where possible. And lets say you write a recursive `rm`, did you remember to check for symbolic links that might take you to `/`? There are a lot of edge cases these tools cover that protect you (https://github.com/coreutils/coreutils/blob/master/src/remove.c - 35 years old and still getting tweaks today, latest patch was June this year, vs your recursive rm function... hrmm).
You should default to using native APIs, then resort to external commands / the shell. Not the other way around :palm: :palm:
All in all, brucehoult's lack of experience in higher level language is clearly showing, which is fine.
You are showing a lack of experience in general system administration tasks on huge datasets and files along with the task of making something that is maintainable by other system admins that may not have your "prowess" in the languages you are using. The right tool for the right job. There is no "order" to things as you put it, assuming so and dismissing these other options as a developer is asinine and just ties your hands unnecessarily.
Which is fine, but it makes your scripting opinions irrelevant in this thread.
Just be a little more careful when dismissing other people's opinion :-DD
:-/O
You do realise you're coming across as super arrogant and hostile here which is why you're getting this kind of reaction from others.
-
I started looking for a way to program the same application in Python and I have found that it is necessary to replicate the operation of standard unix tools like "file", "find", "wc", "grep", etc.
Uh? These are all individual programs. You can call them from Bash, but otherwise from any other scripting language that allows executing external programs, which both Lua and Python can.
Bash doesn't do anything else than spawning processes for any command you ask it to execute and that's something you can do in many languages.
The only thing that may be more or less difficult in other languages is directly piping one executable to another - that would be something to check.
For Python, this is where to start: https://docs.python.org/3/library/subprocess.html
I am familiar with the subprocess lib. I have used it frequently to call some external tools like ImageMagick that sometimes seem to me better than the corresponding Python library (PIL / Pillow).
That's nice to know.
But calling an external program is usually more messy, slower and harder to communicate with than using the language's own tools.
That's what Bash does.
Holy crap, as I said the whole topic was about replacing Bash with another scripting language.
It wasn't about rewriting the whole Linux/Unix ecosystem from scratch.
Of course you can write directly in a given language without resorting to executing outside processes.
That was not the topic at all.
-
Not really, it depends on what you're doing. Try using your native RegEx engine on a huge file for example, you now suddenly need to deal with reading the file as a stream as loading the entire file into RAM would be a huge waste of resources. The complexity of what you're implementing suddenly goes up and it may not be worth the time/effort or cost to implement all the edge case error handing, etc.
No, not "not really" it doesn't matter what you're doing in the grand scheme of things.
you now suddenly need to deal with reading the file as a stream as loading the entire file into RAM would be a huge waste of resources. which is exactly what nodejs was designed for. Streams are one of the common ways to process data in nodejs. So what's the argument? This isn't PHP where everything is synchronous (by default) and everything loaded into RAM.
Not when you're dealing with a huge directory structure and you're coming from a scripted language where it's infinitely faster to use `rm` or `cp`. Again it depends on your use case, you might even want to take advantage of `rsync` to avoid unnecessary copying of files where possible.
Which is a pretty specific situation for a discussion about a scripting language. It doesn't apply to the discussion because it's not a general problem. As far as I know, OP hasn't posted any specific needs that they want to be addressed. So the discussion shall remain about general problems, not specific ones [that happen to play into your argument (as it seems)].
You are showing a lack of experience in general system administration tasks on huge datasets and files along with the task of making something that is maintainable by other system admins that may not have your "prowess" in the languages you are using.
I have a lack of experience in general system administration tasks but that doesn't disqualify me from being a part of the discussion.
The right tool for the right job.
I agree.
There is no "order" to things as you put it, assuming so and dismissing these other options as a developer is asinine and just ties your hands unnecessarily.
Oh, yes. Absolutely. There's an order and you are mistaken for thinking that there isn't.
You do realise you're coming across as super arrogant and hostile here which is why you're getting this kind of reaction from others.
Well, I'm kinda used to that. Doesn't surprise me at all to hear.
You call me "super arrogant and hostile" but brucehoult saying:
Which is fine, but it makes your scripting opinions irrelevant in this thread.
Is totally fine? Look, I don't even want to step on specific things on what others have said.
I'm not offended, not in the slightest.
But I'm surprised to hear that it's somehow ok for someone to dismiss someone's opinion and then getting a "reaction" (so to speak) and that's what you consider "hostile"?
In my defense, you don't get to discredit me without proving you're right.¹ That's how I work, nothing to do with hostility, or arrogance.
It's really sad that so many people fall into the trap of mistaking confidence with arrogance.
I'm not an arrogant person. I have plenty of years of experience in software engineering and if I disagree with someone (like brucehoult) then I'll make my word heard.
It's just not true that you usually call external programs from JavaScript - it's just the way it is. I don't know what else to tell you.
This is also true for other scripting languages such as PHP.
And the reasons are as clear as they can get: it's a messy and slow process - whether you want to believe that or not.
How many people in here do you think have the credentials/experiences to speak about JavaScript? I would like to know.
What I'm telling you comes from experience, not arrogance.
I'm not saying that you shouldn't ever call an external program from a scripting language. What I'm saying is, that this is not how things are usually done in higher level languages. Why? Because the higher level language already provides API for common functionality. You're falling in a similar trap that brucehoult has fallen into. You're showing clear ignorance (not arrogance, but ignorance) to the nodejs / JavaScript Language. That's the whole point, you don't understand how things are done in JavaScript, so how could you possibly argue with someone who knows how it works?? JavaScript has solutions to these problems, you're either too ignorant to know about them, or just too lazy. That's how it is.
If you're using an external program, you better have a good reason for it, otherwise, you're doing things wrong.
Take it from someone who's been programming for over +10 years now.
¹ brucehoult's argument got diminished once I took a look at the implementation of shelljs. Unfortunately for him, it even proved my point. So what exactly do you expect me to do? It should be clear that he's missing the experience to make a meaningful argument in that area when he can't even make a first proper argument. By the way, "Oh, really?" is not a good way of starting your argument.
But whatever, think what you wanna think (people do that all the time anyway *cough* flat-earths). As long as I don't get censored, I'm happy because everyone reading can form the own opinion - that's what matters to me. (this is also why I sometimes write more, because I like to keep a record of things).
-
you now suddenly need to deal with reading the file as a stream as loading the entire file into RAM would be a huge waste of resources. which is exactly what nodejs was designed for. Streams are one of the common ways to process data in nodejs. So what's the argument?
Great, nodejs handles that, doesn't mean that your statement holds true for other languages/platforms which your statement implied.
Which is a pretty specific situation for a discussion about a scripting language. It doesn't apply to the discussion because it's not a general problem.
Performance is always a general problem and concern and if you're not writing your code to be performant then you should re-assess how you write code. I have seen this exact situation countless times, most often with WordPress plugins that write a directory of files for whatever reason, and they give the user a `purge` feature. This purge when run takes so long it causes the php process to hit the max execution time leaving the job 1/2 done. It's far more general then you think.
I have a lack of experience in general system administration tasks but that doesn't disqualify me from being a part of the discussion.
I never said that it did. I am saying that there is a side to software development that developers often fail to heed/understand. The "it's fast enough for me" or "works well enough for me" and not taking into consideration the performance requirements when the application is scaled up to any kind of level causes no end of sysadmin headaches. Just because it's more "proper" in your eyes doesn't make it the more correct way to do things. 30+ years of sysadmin and software engineering across dozens of languages has taught me this lesson countless times as I like you, try my damnedest to avoid using exec/system/popen if possible.
Oh, yes. Absolutely. There's an order and you are mistaken for thinking that there isn't.
There is an order based on the demands of the application and the time/cost budget involved in the development. There is no fixed constant order as you put it.
You do realise you're coming across as super arrogant and hostile here which is why you're getting this kind of reaction from others.
Well, I'm kinda used to that. Doesn't surprise me at all to hear.
You call me "super arrogant and hostile" but brucehoult saying:
Re-read what I said, I stated you're coming across this way, I did not say you are, or intend to be. But the way you continually use emoticons in your posts, making out that your laughing in peoples faces is doing you a disservice.
It's just not true that you usually call external programs from JavaScript - it's just the way it is. I don't know what else to tell you.
This is also true for other scripting languages such as PHP.
Can't speak for JS myself, but I have seen it done in PHP hundreds of times over the years, in some cases warranted (ie, pre-gd2 as I mentioned earlier), and still done today in some instances where there is a need for doing so (ie: try and do a DNS lookup against a specific DNS server without implementing the DNS protocol from scratch).
And the reasons are as clear as they can get: it's a messy and slow process - whether you want to believe that or not.
It depends on if the overheads of fork/exec outweigh the performance benefits of executing a native binary that is faster then your own implementation.
How many people in here do you think have the credentials/experiences to speak about JavaScript? I would like to know.
What I'm telling you comes from experience, not arrogance.
Again, I didn't say you were arrogant, but your method of writing is coming across this way.
Take it from someone who's been programming for over +10 years now.
Ok son, 30+ here, and I am sure there are others here with far more then even I have.
Edit: omg, just saw your signature image... are we really playing that game?
(https://stackexchange.com/users/flair/319887.png?theme=dark)
And I have been pretty much inactive on there for years now. It's not that hard to get tons of rep there.
-
Performance is always a general problem and concern and if you're not writing your code to be performant then you should re-assess how you write code.
Hence why you should AVOID spawning processes. In MANY (not all) but in MANY cases the overhead of setting up a new process etc. TAKES SUBSTANTIALLY LONGER than using native APIs.
I have seen this exact situation countless times, most often with WordPress plugins that write a directory of files for whatever reason
Oh c'mon, WordPress? WordPress is a lost cause and a joke :-DD
This purge when run takes so long it causes the php process to hit the max execution time leaving the job 1/2 done. It's far more general then you think.
PHP's doomed anyway, but that's my opinion.
30+ years of sysadmin and software engineering across dozens of languages has taught me this lesson countless times as I like you, try my damnedest to avoid using exec/system/popen if possible.
So where's the disagreeing part?
Just because it's more "proper" in your eyes doesn't make it the more correct way to do things.
I haven't said anything about "proper" I said it's unusual to do such a thing, not implying that it is improper.
There is an order based on the demands of the application and the time/cost budget involved in the development. There is no fixed constant order as you put it.
Well, I can see where this is coming from. But aren't we talking about someone who wants to switch their personal scripts to a different language?
I get what you mean, in a professional context this is definitely the case, no doubt. But I don't think it applies to "business" outside the job, does it?
I mean, I'm doing an assumption here, but aren't we all enthusiasts of electronics/programming etc. who want to do their best?
Excuse me if this should be taken in a professional context, because then I'm definitely wrong - I can see that.
I just think when you're doing things voluntarily these factors (time/cost budget) don't play a huge role.
It might for some, but not for me. And I'm pretty confident in saying that most hobbyist do it for the fun, not because they're on a "budget" or whatever.
I'm a hobbyist (as my profile signature tells you), I've worked professionally as a Software Engineer, but I don't think I want to go back into that hell :scared: 😂
Re-read what I said, I stated you're coming across this way, I did not say you are, or intend to be. But the way you continually use emoticons in your posts, making out that your laughing in peoples faces is doing you a disservice.
I'm sorry, I just find some of these statements to be ridiculous and I have no shame of hiding that. It's not meant in a demeaning way, I just want to show people when I'm not taking them seriously - that's all.
Ok son, 30+ here, and I am sure there are others here with far more then even I have.
Doesn't mean you have the knowledge :P I know, I'm probably going into dangerous terrority here but just because doing things for +30 years doesn't mean you know better than someone who did it for less time :-DD Technology has quite improved, and nowadays, it's actually a disadvantage because younger (myself excluded) people actually do better because they got to learn technologies that were impossible back in the time - I don't think this trend ends here. Plus, nowadays it's much easier to get to learning resource than it was 20 years ago. Not a personal attack, just saying, experience is not everything.
And please, could you not call me "Son"?
Off-topic: Man, quoting people here in the forum is a real pain :S
Edit: omg, just saw your signature image... are we really playing that game?
I don't play games, but it's NICE to see that you do :-DD :-DD get lost.
-
Doesn't mean you have the knowledge :P I know, I'm probably going into dangerous terrority here but just because doing things for +30 years doesn't mean you know better than someone who did it for less time :-DD Technology has quite improved, and nowadays, it's actually a disadvantage because younger (myself excluded) people actually do better because they got to learn technologies that were impossible back in the time - I don't think this trend ends here. Plus, nowadays it's much easier to get to learning resource than it was 20 years ago. Not a personal attack, just saying, experience is not everything.
The younger generation that were not exposed to the low level details of how the computer architecture actually functions usually end up writing bad/poor code that we then spend inordinate amounts of time trying to fix. And while I agree entirely that experience is not everything, it is a far better teacher then just reading. Even still I make full use of the learning tools I have available and work on projects most would dream of being a part of. These days most of my free time is spent on the Looking Glass project which is cutting edge technology that I made possible.
Your prior post spoke in such a way to state that your 10+ years of development experience means we should just trust your word as gospel. All it did in my eyes was show me how much you still need to grow as a person and why I called you "son". Which btw, is not an insult
In American English, it is acceptable and common that an older man calls a man his junior, "son"—even if the younger man is not the older man's child (or related to him in any way). Used by an elder person as a form of address for a boy or young man
You publish your age, and your experience in this field, that by definition makes you my junior.
Edit:
That said, I am done, think what you like, but please stop acting like you know better then everyone else. Speaking down to others by laughing at them with the emotes as you did is only going to result in threads like this repeating over and over again.
-
Your prior post spoke in such a way to state that your 10+ years of development experience means we should just trust your word as gospel.
No, that wasn't the intent and I wouldn't advocate for such a lunacy. Nobody's words should be taken as gospel.
What I was trying to do was to give my word some credibility, since it seems like people think I don't know what I'm talking about.
All it did in my eyes was show me how much you still need to grow as a person and why I called you "son".
Well, good for you then. I can see how I need to grow as a person, especially from someone who takes the bait and think I'm playing games like a little child - I'm not. You fell for it, and now ... I think (actually) lesser of you, because I'm indeed not playing games.
YOU assuming I am, really tells a lot more about you than me ANYWAY we're going HUGELY off-topic and I will leave it AT THAT.
All it did in my eyes was show me how much you still need to grow as a person and why I called you "son". Which btw, is not an insult.
You don't get to call me "Son" because your not my dad, simple as that. Nothing to do with being offended or whatever.
OP: sorry for getting into "DRAMA" :-DD I'll keep my mouth shut.
I'll refrain from using the laughing emoticons to my closing statement to this discussion.
That said, I am done, think what you like, but please stop acting like you know better then everyone else.
I will (think what I like) and I will act to know better until there's someone who can prove me wrong. Nobody's on this thread so far convinced me otherwise. You have a problem with that? Fine. It's your problem then.
-
Well, good for you then. I can see how I need to grow as a person, especially from someone who takes the bait and think I'm playing games like a little child - I'm not. You fell for it, and now ... I think (actually) lesser of you, because I'm indeed not playing games.
Ok... so trolling, a bannable offence. Careful now
-
Oh, really?
var shell = require('shelljs');
if (!shell.which('git')) {
shell.echo('Sorry, this script requires git');
shell.exit(1);
}
// Copy files to release dir
shell.rm('-rf', 'out/Release');
shell.cp('-R', 'stuff/', 'out/Release');
// Replace macros in each .js file
shell.cd('lib');
shell.ls('*.js').forEach(function (file) {
shell.sed('-i', 'BUILD_VERSION', 'v0.1.2', file);
shell.sed('-i', /^.*REMOVE_THIS_LINE.*$/, '', file);
shell.sed('-i', /.*REPLACE_LINE_WITH_MACRO.*\n/, shell.cat('macro.js'), file);
});
shell.cd('..');
// Run external tool synchronously
if (shell.exec('git commit -am "Auto-commit"').code !== 0) {
shell.echo('Error: Git commit failed');
shell.exit(1);
}
Yes, really. shelljs does not use the shell, it has a misleading library name and you fell for it. :-DD
shelljs is just a wrapper around native JavaScript APIs like the fs module etc.
For example, shell.sed does indeed NOT call the sed binary but instead emulates the expected behaviour with native JavaScript.
*Some* of them. Just like busybox emulates many of the same programs. Nothing wrong with that.
Bash itself emulates/reimplements some commands that are available as external programs in /bin or /usr/bin (and which you can use instead if you invoke them with the full path).
Do you think shelljs emulates that call to git?
Because that's a big "nope".
-
Take it from someone who's been programming for over +10 years now.
Ok son, 30+ here, and I am sure there are others here with far more then even I have.
A 27 year old lording it over everyone because he has 10+ years programming experience? Meaning he's starting counting from somewhere in the middle of high school.
Laughable, isn't it?
Fuck me. I was programming an Apple ][ in Pascal and machine code and wrote my own Forth on it in 1980. And programmed TI and HP calculators before that. I was using LISP on a VAX in 1982. I've heard that's a high level language. I've worked at Mozilla improving the Javascript JIT in FireFox -- when MMMarco was 13. But, yeah, I've got no experience in Javascript. I've won multiple prizes in the ICFP (International Conference on Functional Programming) 72 hour programming contest, using the high level language "Dylan". But apparently I don't have any experience in high level languages.
I could go on, but I won't.
-
I could go on, but I won't.
I for one trust your real world experience, even if I sometimes disagree, and (inadvertently!) apparently annoy you.
Note: not the opinion, but the way brucehoult often explains the basis of their opinion based on practical experience. Sometimes it takes a bit of prodding, which seems to annoy some members here. I love it, because that way I can trust, but also test and verify. Reality beats theory, and experience has utility and value, but reproducibility and verifiability is king.
Me, I have a rather spread-out professional history; ridiculous, really. (As in, I would be sceptical if someone claimed a similar one, it's just so.. unlikely. It's just random chances in life, though.) I don't have the kind of spear-tip experience as say brucehoult has, as I am more of a generalist and synthesist, relying on the width/spread of my practical experience to give hints and guidance. My education is in computational materials physics, but I've also taught a couple courses at uni (basic it courses and web stuff, back around the turn of the century), and ran my own IT company for a few years (profitably, but completely unsuitable to my personality). I am analytical to the core, which requires me to try and explain the reasons for my opinions instead of stating them as fact, or relying on myself or my background to be any kind of authority. I often have wrong opinions myself! My scientific background has taught me provability and reproducibility trumps opinions or beliefs, and even previous experience. Otherwise, our sciences would not – could not – progress at all.
I love hanging out here, reading the discussions where experienced people in their own fields explain why and how projects can be made better.
It just annoys me to no end when people confuse opinions with facts (≃ experimentally verifiable details and results). Sometimes when I prod into it, into the reasons behind the opinions, people see it as questioning their experience, but it isn't that, it is just inquisitiviness and trying to get to the reasoning behind the things at hand.
Even in the programming threads, I often learn more, because the stock of human knowledge about our tools well exceeds what a single person can learn in a lifetime of personal experience. I will never know enough; my education is never complete. Many times, when someone poses an interesting question, I have no idea what the answer is, but I happen to know how to find out, so do that, and show the results and how I did it. I used to do that at stackoverflow (https://stackoverflow.com/users/1475978/nominal-animal) and math.stackexchange.com (https://math.stackexchange.com/users/318422/nominal-animal), but their social rules became impossible for me to follow. (Not ideologically; it is literally too much work for me to keep track of the social rules there. Plus, fundamentally, popularity does not guarantee or even correlate reliably with correctness, so the entire voting up good answers is a popularity game only. Social gaming.)
On the electronics front, although I do have some formal education in it, I am still a newbie hobbyist after a decade or so. I do try to help others with e.g. suggested schematics and even boards, but I make sure they're aware I'm only a hobbyist and not experienced in the practical electronics side. Heck, BGA packages still scare me away. There is nothing wrong, nothing demeaning, nothing belittling in admitting one is still learning. Many seem to feel so, but to me, it just means someone is willing to take the effort to learn, and listen to suggestions and criticism, which is a HUGE positive in my mind, because that attitude works. Pride, and believing one already knows everything worth knowing, does not work in practice.
I openly admit I'm still addicted to problem solving: finding verifiable, effective, robust solutions to tasks that are considered impossible or too difficult.
Finally, I am fully aware that I am way, way too verbose with my walls-of-text responses. Apologies; I am working on it.
-
Of course, if your "software engineers" believe only monolithic and fully proprietary applications are the only sensible way to construct applications, they will never accept that or say using user-specified external programs or scripts to work on that user's data. After all, all data belongs to the application, not the user, eh?
We aren't even on the same page mate. I don't disagree with you either. It's just old hat and mostly what the entire 'enterprise software industry' has been moving away from at rapid pace.
If I go on a call right now with a customer to talk about a solution. For a fact. I am NOT going to recommend a kernel bound, native C++ application set and scripts, pipes, sockets and devs. Why? Because I sure as hell don't want to maintain it, or make the support teams suffer the same.
C++ developers are like hens teeth and their business application development rate is pathetically slow. So the industry moves to where there are developers. Java, Python, even things like Go! The places you will still find C, C++ is in the low-level, high speed components in the likes of stock exchanges and realtime marketdata processing systems like Algos.
On "target platform". It's very seldom I will develop an application with a target of a bare OS. Extremely unlikely actually. At very least it will be targetting a pre-determined landing environment provided as a service AND running in some form of virtual machine or container. These days its more likely to be Dockerized at build time, encrypted and stored in a repo, then deployed by someone else onto a logical pod in a multi-zone international cluster.
Most applications are stateless. Most applications amount to micro-services. Persistence is handled in many different ways, but very, very, very rarely files or even "OS level" filesystems. Transient state, distribution, IPC et. al. is handled in some form of middleware. Deployments are full automated, applications can be scaled elastically on demand. Version rollout and rollback etc. etc.
Even when I was developing kernel bound "RHEL5" applications such as the Mexico Equities Exchange (BMV) Common Customer Gateways which where rated for 20k messages per second, per session, 20 sessions 20uS max latency, the deployments, configuration and maintaince was "platform" based. The customer took our build, applied their own configs via a dynamic configuration platform and then man handled the binaries onto "pre-canned image based" hosts and started them via remote process management agents, like AutoSys, Tidal etc.
Again. When you have one person writing, testing, deploying an application you are not experiencing what modern software engineering is all about. It's all about running away from the model you describe, due to it being unmaintainable and unscalable.
I'm not saving you can run a system completely script free. It's that Application development has completely separated itself from such fragile and mundane tasks as reinstalling an OS every ''''ing day and fiddling with cryptic spehegetti devops scripts. Instead we heir devops people to do that while writing systems to avoid needing them either.
Have an application processing 100k requests per second and it's struggling? Click'ity click, now you have 1000 edge nodes running hte application in 14 different regions. Demand has suddenly stopped? Would be expensive to maintain all those OS's and scripts. Click'ity click BOOM! 900 of them are deleted out of existence a matter of seconds later.
I should be obvious I am not a front-end developer. I have written one or two GUIs, but they are so... brittle and slow to develop, WUIs or webUIs being much faster are more common and due to that one type of service I have written a lot is an API front end for said WebUIs. Often REST, Kafka etc.
EDIT: As to how I feel about the criticism. The important point is... you don't pay my salary. ;)
I am also 10% trolling. You know that right. I'm basically standing up a model enterprise 2020+ architecture at the bleeding edge of massively scaled enterprise applications running on some of the biggest clusters and data centres in the world against your 1960s Unix modular 'native binary coupled' architecture.
It's not fair to compare them really. We also do use Linux at the core of 99% of the systems when you peel back all the layers you eventually come to a linux kernel running in a supervisor somewhere. Those OS's need developing. Yes there are scripts involved there. However, even Linux itself has been moving away from that model over the past decade or so.
-
If I go on a call right now with a customer to talk about a solution. For a fact. I am NOT going to recommend a kernel bound, native C++ application set and scripts, pipes, sockets and devs. Why? Because I sure as hell don't want to maintain it, or make the support teams suffer the same.
Now you're mixing technical choices with business reasoning. Your business is not the only business out there, you know.
I'm not saving you can run a system completely script free.
But that is exactly what this entire thread is about: systems that DiTBho is working on, have very strict reliability requirements, and has found the necessary interface glue between the system's applications and services to be a weak point, requiring something better (easier to maintain, debug, and verify/validate to required standards).
Just because you wouldn't be able to make a buck doing things that way, does not mean it is not a perfectly valid business niche for someone else.
EDIT: As to how I feel about the criticism. The important point is... you don't pay my salary. ;)
I have walked the business path, and decided it is not for me, because I do not want to be ruled by financial concerns, to evaluate solutions based on their costs.
That does mean I am much better suited for research, for developing new technologies that others may later adapt to solutions that lead to profitable business, but without people like me kicking the butt of the "just throw more developers and hardware at it, and it'll work out" crowd, you'd drown in unreliable shit within a generation. It has happened before, it will happen again.
I am also 10% trolling. You know that right. I'm basically standing up a model enterprise 2020+ architecture at the bleeding edge of massively scaled enterprise applications running on some of the biggest clusters and data centres in the world against your 1960s Unix modular 'native binary coupled' architecture.
Do you realize, at all, that you are trolling exactly the people who have developed and still develop the underlying transports and mechanisms that makes your lofty business possible in the first place?
However, even Linux itself has been moving away from that model over the past decade or so.
And the developments like systemd that you think are making it better suited for your use cases, are in reality moving it away.
Fifteen years ago, I set up a small test cluster (on a few unused old desktops), and did some experimentation on ksplice (live-patching), kernel-assisted userspace checkpoint-restore (for hot reconfiguration of cluster nodes to avoid stopping long-running distributed tasks), and various imaging approaches to speed up compute node reinstallation. This worked, but since it wasn't done elsewhere, it was too much to ask from the other sysadmins, and was rejected from wider use.
Today, dbus interconnections are a source of single point failures, which makes userspace checkpoint-restore unfeasible. Auto-reconnection of the connections without additional userspace code is just not possible. Systemd itself is one huge risk of single point failure, and has revealed years-long security holes, because its developer base is tiny and insulated from the rest of the development. Using it makes your systems fragile, because any unexpected failure of a service daemon can destabilize the systemd daemons and cause the system to become unoperable. It is very difficult to track down the exact cause, because such cases only reveal the services that failed, and the logs never point to systemd as being the cause, unless you check (whether its service daemons still operate correctly, or are in an inoperable state), which basically nobody does. This, and its viral subsumption of unrelated services, is what drives the hatred towards systemd, not some silly "we want to keep things like they were in the 1960s".
And unless you pay, no more live patching either: you will be rebooting your nodes for kernel updates. You'll need to reboot anyway when systemd is updated, so who cares, right?
I have a view further in the future than you do. You think it means I'm "stuck in 1960s". No, I just want to produce something better, without repeating past mistakes and the oh so popular "this is new, so it has to be better!" belief by most youngsters. We've seen too many of these come and fail already to get excited by every single new and shiny thing.
In many threads, I've described how I believe distributed and parallel computing will be the way forward, but our current programming languages just don't have the proper abstractions that would facilitate that effectively in practice. Sure, there are all kinds of CS theories out there, experimental and academic languages, and some even backed by huge corporations, but they are almost all the same-old, same-old; nothing that transported back two decades would surprise developers at all compared to the advances in hardware capabilities. I know, because I just cannot help myself and keep an occasional eye at ACM and elsewhere.
I hope you realize you have been arguing a business case against exactly the people who make your business possible in the first place. It is unlikely, though, because people who rely on complex abstractions to get their work done, often lose the ability to understand the reality behind the abstractions. Perhaps you are an exception, but I doubt it; your arguments are all just based on your own opinions and experience, with nothing verifiable or testable in them, which indicates you're not actually that interested in the factual reality beyond your business needs.
Which is fine, really, except I'm not interested in interacting with such people at all. I want to keep doing the grass-roots level that hopefully will help/lead to a true step forward, ending the programming language stagnation I've seen cripple software development for the last quarter century. It has been papered over by the huge advances in computing capacity, but that just makes it wasteful. With the current laughably bad reliability and security record, we need, and deserve, something better. It may not make business sense to try to do that, but that is the kind of change I'd like to be part of in this world.
-
Do you think shelljs emulates that call to git?
Because that's a big "nope".
That's literally the ONLY thing that shelljs ISN'T EMULATING - clear that you would jump onto that.
A 27 year old lording it over everyone because he has 10+ years programming experience? Meaning he's starting counting from somewhere in the middle of high school.
I'm not lording over anything, if anything, you're lording over me. You're wrong - I (somewhat) proved that you're wrong and it is you, who cannot deal with that. Sure, you might be a very experienced assembly/low level/C++ whatever programmer, but that doesn't make you a good JavaScript Developer.
I assume that you're not, because "calling external programs" is not something USUAL you do in JavaScript - that's the whole point of my argument, and always has been.
You said my opinions were irrelevant, because "scripting" would involve calling external programs.
Which is not true and why I disagree. That's the whole argument right there.
I don't know about you, but if you're telling a person (in person) that their opinion is "irrelevant" you better prepare for a fight if you got someone in front of you who's proud and passionate about the topic at hand. You provoked this whole mess.
Your one and only argument "shelljs" is laughable because it takes seconds to disprove your whole argument.
I really don't know what you think I should do?
You fucked up, own up to it ;)
Fuck me. I was programming an Apple ][ in Pascal and machine code and wrote my own Forth on it in 1980. And programmed TI and HP calculators before that. I was using LISP on a VAX in 1982. I've heard that's a high level language. I've worked at Mozilla improving the Javascript JIT in FireFox -- when MMMarco was 13.
:-DD yeah I get it, you're a "big dog" doesn't mean you have the experience in JavaScript.
But, yeah, I've got no experience in Javascript.
I bet you don't.
But apparently I don't have any experience in high level languages.
Yes, you don't. Get over it.
-
:-DD yeah I get it, you're a "big dog" doesn't mean you have the experience in JavaScript.
:palm:
I've worked at Mozilla improving the Javascript JIT in FireFox -- when MMMarco was 13
But apparently I don't have any experience in high level languages.
Yes, you don't. Get over it.
:palm:
I've won multiple prizes in the ICFP (International Conference on Functional Programming) 72 hour programming contest, using the high level language "Dylan"
@MMMarco I warned you against trolling, and you continued, you're done here. Take a two week break. Hopefully when you return you have learned to show some respect towards others and take the time to actually read what people are saying before responding in such an asinine way.
-
Do you think shelljs emulates that call to git?
Because that's a big "nope".
That's literally the ONLY thing that shelljs ISN'T EMULATING - clear that you would jump onto that.
The git call was the entire point of the example I posted. The other stuff was just for context, showing it was a real example, not just something I was making up on the spot.
I just spent 18 months working as a JS dev at a small NZ web-based startup "TIME" listed in their "100 Most Influential Companies of 2022" (in the world).
The RISC-V world is exploding right now in a way it wasn't two years ago, so I'm jumping back into that now.
-
bash is feeling rather long in the tooth and has implemented a number of questionable decisions (including one that led to an exploit named "shellshock": it automatically executed code found in environment variables). But the concept of the Unix shell doesn't begin and end with bash. Zsh is a lot more modern for example. And there is Scsh!
-
bash is feeling rather long in the tooth and has implemented a number of questionable decisions (including one that led to an exploit named "shellshock": it automatically executed code found in environment variables). But the concept of the Unix shell doesn't begin and end with bash. Zsh is a lot more modern for example. And there is Scsh!
Interestingly both Bash and Zsh were released at about the same time.
I don't know much details about Zsh. What does it offer that Bash doesn't?
I think the OP was looking for a scripting language a bit more elaborate than just a "shell". But maybe Zsh is? In all cases, you're right in pointing out that there are alternative shells out there.
The thing is, many Linux tools rely on Bash - like 'autotools' (can it run with anything else?).
Which, btw, means that if you're using a scripting language, you may still need to have Bash available to execute some existing tools.
So a question: can you use autotools say with Zsh?
-
There are a lot of small differences, making a comparison list would be pretty hard (although no doubt someone on the Internet has already tried; I have not looked). In general the philosophy of Zsh is not to be quite as constrained by compatibility with older shells. Bash on the other hand (as you can sort of tell from its name: "Bourne-Again Shell") is required to be compatible with Bourne shell scripts as in many systems it is linked to /bin/sh.
Some of the interesting features are nested substitutions, process plumbing, dynamic directory names, catch and throw, evaluation hooks, and precompiled batch files.
About autotools, I do not know for sure. It may have bashisms.
-
I think I've found an answer to this: https://www.gnu.org/software/autoconf/manual/autoconf-2.67/autoconf.html (https://www.gnu.org/software/autoconf/manual/autoconf-2.67/autoconf.html)
Zsh has a compatibility mode that you need to use for this. So, that should be workable.
The doc says that Mac OS X switched from Zsh to Bash. Would be interesting to know why. Possibly they were running into too many compatibility issues with existing stuff.
-
MacOS switched back to zsh as the default shell in a later version. I think it was Catalina (10.15, October 2019).
-
MacOS switched back to zsh as the default shell in a later version. I think it was Catalina (10.15, October 2019).
This of course is only for new accounts. Old accounts keep their existing settings which, for me, have been bash ever since Rhapsody, which IIRC had tcsh as the default shell.
-
MacOS switched back to zsh as the default shell in a later version. I think it was Catalina (10.15, October 2019).
This of course is only for new accounts. Old accounts keep their existing settings which, for me, have been bash ever since Rhapsody, which IIRC had tcsh as the default shell.
Any ideas why they have kept changing?
-
MacOS switched back to zsh as the default shell in a later version. I think it was Catalina (10.15, October 2019).
This of course is only for new accounts. Old accounts keep their existing settings which, for me, have been bash ever since Rhapsody, which IIRC had tcsh as the default shell.
Any ideas why they have kept changing?
Nope. New people?
-
Yes, maybe just the preference of a given manager at a given time. I was just wondering whether there could be definite technical reasons for changing.
-
Yes, maybe just the preference of a given manager at a given time. I was just wondering whether there could be definite technical reasons for changing.
From Rhapsody to now is 25 years ...
Most sources say OS X always used bash and zsh is the new thing. I couldn't immediately find a list of default shell by version. But this O'Reilly book confirms that tcsh was the default shell as recently as Jag-wire (10.2) -- and my memory is that was the same for all previous versions, back to NeXTStep.
https://docstore.mik.ua/orelly/unix3/mac/ch01_05.htm
-
Which is fine, really, except I'm not interested in interacting with such people at all. I want to keep doing the grass-roots level that hopefully will help/lead to a true step forward, ending the programming language stagnation I've seen cripple software development for the last quarter century. It has been papered over by the huge advances in computing capacity, but that just makes it wasteful. With the current laughably bad reliability and security record, we need, and deserve, something better. It may not make business sense to try to do that, but that is the kind of change I'd like to be part of in this world.
I honestly don't think you understand that scale. It's all very well saying it's wasteful, but honestly, again, I don't think you understand it.
I've read your walls of text, mostly. It's all out of date in terms of software development. It's all narrow minded "I've been doing this my whole life" small scale crap.
Sure it's needed at times, but honestly, I'm not tricking you, we do everything in our power to get away from that bullshift hardcoded cryptic nonesense.
If you want a main driver for this, beyond anything else.... go and source yourself 100 developers for software project.
Now come and tell me what language and platform are you going to develop it in?
-
I think the OP was looking for a scripting language a bit more elaborate than just a "shell"
Yup, I confirm :D
-
If you want a main driver for this, beyond anything else.... go and source yourself 100 developers for software project.
It appears that the meaning of "The Mythical Man-Month", after 50 years, has yet to be learned.
-
I think the OP was looking for a scripting language a bit more elaborate than just a "shell"
Yup, I confirm :D
Python is the tool you are looking for.
-
The end of this thread is getting particularly funny.
-
If you want a main driver for this, beyond anything else.... go and source yourself 100 developers for software project.
It appears that the meaning of "The Mythical Man-Month", after 50 years, has yet to be learned.
I barely restrained myself yesterday from commenting that I have never seen a project that needed 100 developers that wasn't a complete fuck up before it even started -- and the only place they happen is typically some "Big Four" consulting company hoodwinking a government department into spending a billion dollars on an IT project that everyone with a clue knows could perfectly well be done for 10 million dollars with a team of 10.
No one in the commercial world uses teams of 100 people working on the same codebase.
-
No one in the commercial world uses teams of 100 people working on the same codebase.
That's an interesting claim. I have a counter-claim that many large tech companies do exactly that: https://en.wikipedia.org/wiki/Monorepo#:~:text=shared%20codebase,%20very%20large%20monorepos
-
No one in the commercial world uses teams of 100 people working on the same codebase.
That's an interesting claim. I have a counter-claim that many large tech companies do exactly that: https://en.wikipedia.org/wiki/Monorepo#:~:text=shared%20codebase,%20very%20large%20monorepos
"the code for a number of projects is stored in the same repository"
i.e. different codebases.
-
The FAANG SV companies are anticompetitive monopolies, not "the commercial world".
Besides, the structure of a version control system is not pertinent to the scaling issues of software product management.
-
I think the OP was looking for a scripting language a bit more elaborate than just a "shell"
Yup, I confirm :D
Python is the tool you are looking for.
I'm currently moving from bash to lua, I wrote about it a few pages ago, and only recently added that I am also writing some "helpers" in C, compiled to help bash scripting, and used as a library for other C helpers/tools, or imported into Lua scripts.
hybrid solution for now
"emerge" is written in Python, I am re-writing(1) it in C because Python has too many critical dependencies, eats too many CPU cycles, and has too much ram on my routers (it's a serious problem when the ram is <256Mbyte and the CPU clock < 400Mhz)
(1) only essential functionalities
- dependency analysis(2)
- calculating the correct order of the packets to emerge
- calculating the correct order of packages from un-emerge
- panic() in case of cyclicality, so I always solve it by hand
- interpretation of Portage recipe classes
- interpretation of the ebuils, from EABI=1 to EABI=9
- so I support all things that are "deprecated and removed" (1..4) to Portage/emerge 2023
(2) I am re-using this algorithm to calculate the order of starting and stopping /etc/init.d services
one algorithm, several use-cases, and re-using code is a very good point!
-
If you want a main driver for this, beyond anything else.... go and source yourself 100 developers for software project.
It appears that the meaning of "The Mythical Man-Month", after 50 years, has yet to be learned.
I barely restrained myself yesterday from commenting that I have never seen a project that needed 100 developers that wasn't a complete fuck up before it even started -- and the only place they happen is typically some "Big Four" consulting company hoodwinking a government department into spending a billion dollars on an IT project that everyone with a clue knows could perfectly well be done for 10 million dollars with a team of 10.
No one in the commercial world uses teams of 100 people working on the same codebase.
Nobody said the same "codebase". If by "codebase" you mean the configuration layer of the project, aka git.
Last project I left had 1039 people on it. Yes, 1039. In about 14 different countries, including, but not least, Russia and Ukraine... at the same time.
Not 1039 developers. Probably about 40% tech. Something in the order of around 200 "code bases" in about 5 git repos.
"codebase" in this sense could be anything from an Oracle distributed configuration and control framework, a kafka serialization frameworks, table compaction jobs, maintenance jobs, ingestion jobs, transformation and enrichment jobs, batch jobs, streaming jobs, micro-services..... WebUIs, data warehouses, data scientist clusters, GPU in memory database systems. And all the security, command/control/reporting, "infrastructure as a service" required to back it all into a somewhat coherent and uniform fashion.
We migrated around about 150 'data feeds' for an international company from one platform, entirely to another. Current feed and historical from around 100 different external and internal 'vendors'/'sources'.
"We" .. my part was about 5 different data feeds, including some in the order of 300million records and day and a datasize in memory of around 250Gb. Others were near-realtime streams of data from external providers arriving in 30 second micro-batches containing 1000-2000 messages. Others where just opening a CSV file and setting it down again in a different format, like parquet.
Writing code and piping things between processes on a CLI.... LOL
EDIT... and yes, I left because it became a "cluster fuck".
It's easy to say, "that's because it was too big".
However, saying that all software can be written by a single person or a tiny team is also easy to say and catagorically false.
Software Engineering today is 90% about that problem and 10% writing code or SQL.
-
On that scaling and this comment:
It appears that the meaning of "The Mythical Man-Month", after 50 years, has yet to be learned.
It doesn't work like that, but if your point is that it doesn't scale linearly, then thank you captain obvious. This is also part of every single software engineering degree you will take.
In software and I imagine most or all engineering communication is key.
Electronics and hardware people know this. If it's just you making the circuit and product end to end, you can skip a VAST amount of diagrams, documentation and specifications. You can skip a vast number of meetings too. You can develop such a product (usually a prototype) amazingly fast.
However, when you can no longer be the single soul person doing the work end to end, the amount of documentation and the details it must go into go up non-linear with the more subdivisions of the work. The number of meetings and workshops and debates and changings of minds and requirements go up and up and up.
In software engineering when a task/system/requirement becomes too big for a single person, you can move to a two man team (Pair programming). However, social and civil issues arise from a team of 2. It's a marriage and will suffer similar issues to one. Often burning out in arguments after 6 weeks or so. So 3 is usually better as no vote can hang without abstention.
3 developers rarely produce 3 times the work of 1. The finger in the air rule in 60% per additional. So 2 devs do as much work as 1.2 devs. 3 about 1.8.
The issue is the communication channels between team members and the inter-coupling of their work requiring communications multiply geometrically. When there are 2 devs, there is only one pathway. When there are 3 devs there are 3. When there are 4 devs there are 6.
Honestly when you get to about 4 devs, you need some management or team leadership and they do not count directly towards teh "man month" count although they do tend to stop "pulling in many directions" problems.
The "preferred" "modern" model is to divide groups of devs up into teams of 5-10 with a "scrum master" serving one or two of these teams and a project manager over-seeing 5 related teams. Software is broken down into "Epics" by management, then into smaller one to three day tasks which are then ordered by priority and dependencies, but ultimately left for developers to pick up and assign to themselves.
This is the development team manages and owns the development.
No. It doesn't work with pure business types. They simply cannot understand that we can tell them "what" they will have but not when, OR, we can tell them when they will have it, but not what.
When you do get through to a team of business types and they trust you, they are usually happy for it. Far, far too often business analysts and customer managers will UNDER estimate just what can be delivered with modern software tools and techniques. They under sell the likely output and enforce stupid requirements resulting in inherent functionality being lost.
Instead by operating very rapidly on a 2-3 week cycle and then demonstrating functioning software each cycle and allowing customer feedback each cycle allows a lot of the otherwise, "Would you just open you eyes and LOOK instead of talking crap!" to be skipped and wasted effort avoided.
Spending 2-3 weeks on an idea and having the customer say, "Nope, absolutely not!". Is far better than doing it on a waterfall for 3 years like civil-service projects where, under their certified methodologys, if a requirement turns out to be wrong after 2.5 years the are legally required to go back to square ONE. This is why many, many government IT projects cost so fucking much it's got nothing to do with software engineering, it's high brow legal quality and control methodologies that require every single line of code be documented. Which is fine if you are writing a 777 autopilot control software, but not a fucking "Apply for your vehicle tax form", only form!
There is no man month. Most tasks are assigned and estimated on a individual basis. Either based on days or just on "complexity". I've been in teams where the estimates where facial expression emojis! Having management constantly under-estimate those aggregates is only because management are ... well management. aka, usually skill-less.
Like everything, it depends where you are and who you are working for and with. Projects come and go, stuff changes. New skills are required, you are expected to learn. Tech lead wants to switch from SpringBoot to a random app container framework you have never heard of.... best get googling.
-
my super-heroes here are Dominic Giampaolo and Cyril Meurillon, who in 1996 designed, implemented, tuned, documented, and tested, and fully debugged, one of the best innovative filesystems ever written!
All alone, only those two guys at Be Inc :o :o :o
-
I think the OP was looking for a scripting language a bit more elaborate than just a "shell"
Yup, I confirm :D
Python is the tool you are looking for.
I'm currently moving from bash to lua, I wrote about it a few pages ago, and only recently added that I am also writing some "helpers" in C, compiled to help bash scripting, and used as a library for other C helpers/tools, or imported into Lua scripts.
hybrid solution for now
"emerge" is written in Python, I am re-writing(1) it in C because Python has too many critical dependencies, eats too many CPU cycles, and has too much ram on my routers (it's a serious problem when the ram is <256Mbyte and the CPU clock < 400Mhz)
(1) only essential functionalities
- dependency analysis(2)
- calculating the correct order of the packets to emerge
- calculating the correct order of packages from un-emerge
- panic() in case of cyclicality, so I always solve it by hand
- interpretation of Portage recipe classes
- interpretation of the ebuils, from EABI=1 to EABI=9
- so I support all things that are "deprecated and removed" (1..4) to Portage/emerge 2023
(2) I am re-using this algorithm to calculate the order of starting and stopping /etc/init.d services
one algorithm, several use-cases, and re-using code is a very good point!
Sorry, I missed reading that you were working in an embedded environment.
In that case Python is oversized and Lua has a lot of advantage.
-
I started looking for a way to program the same application in Python and I have found that it is necessary to replicate the operation of standard unix tools like "file", "find", "wc", "grep", etc.
Uh? These are all individual programs. You can call them from Bash, but otherwise from any other scripting language that allows executing external programs, which both Lua and Python can.
Bash doesn't do anything else than spawning processes for any command you ask it to execute and that's something you can do in many languages.
The only thing that may be more or less difficult in other languages is directly piping one executable to another - that would be something to check.
For Python, this is where to start: https://docs.python.org/3/library/subprocess.html
Or just goto pypi, a wrapper for subprocess for calling linux commands directly in python
https://pypi.org/project/sh/