Python becomes the most popular language

[tszaboo]

This is an electronics forum, which is one of the areas where Python is not as widely used due to its speed. This is very noticeable in the comments, always obsessed with speed, as if speed were the only parameter of importance.

Yeah, only the
- ERP system for the production
- the product testing automation
- release scripts for the FW
- and the HW
- all the machine learning
is running on python where I work. The backend and the website hosting not yet because hiring JS developers is still easier.
Python might be slow, but while people who never actually used it write these highly opinionated long posts, that it's bad because -insert reason here-, using the same time we just write scripts that gets the job done.

[nctnico]

Server side is always a highly constrained / defined environment. For desktop use the picture is entirely different; Python is a huge PITA for desktop software distribution.

[voltsandjolts]

Your previous comment stated 'commercial use', seems that you specifically meant desktop commercial apps.

I agree somewhat with that, desktop app distribution is not slick, like java .

[brucehoult]

It's possible Java and separately JavaScript might lie somewhere between the development time of C++ and Python -- Java closer to C++, and JavaScript closer to Python, and bot run maybe 20% slower than C++, not 20x slower.

I don't agree. I ported a library from pure C to pure Python and for sure it's 20 times slower.

So, in fact, you are agreeing with me.

[brucehoult]

This is an electronics forum, which is one of the areas where Python is not as widely used due to its speed. This is very noticeable in the comments, always obsessed with speed, as if speed were the only parameter of importance.

Of course speed is not the only parameter of importance. No one claims that.  But it *is* important. For any given application there is "fast enough" and "not fast enough" and C is on the correct side of that line more often than Python is.

Many things are fine running on Python on a 4 GHz processor, but they'd be just as fine running in C on a 200 MHz processor. Those are a lot cheaper, and use a lot less energy.

[Cerebus]

Forget speed for a moment, what about efficiency? Quite how much physical power do you use if you choose to do identical tasks in native Python (or anything equally inefficient) rather than a typical 3GL? How many more servers do you need to handle your workload? How much more air-con to keep them happy?

The limiting factor in new data centre builds at the moment increasingly seems to be local electricity capacity. Even if you're minded to ignore the environmental impact of rolling out data centres at the rate we are at the moment, pure cost and the fact that it's becoming harder and harder to find somewhere  where it's practicable to build one are going to make efficiency a more important consideration again.

[SiliconWizard]

Yep, software devs have been spoiled by abundant and relatively cheap, while increasingly powerful hardware for a few decades now, but if the trend shifts back toward more efficient programming, it will be for energy efficiency reasons, and you can bet your ass that this is going to be a definite concern soon enough. Heck, energy efficiency already IS a huge concern, and if software is more or less "left alone" for now, it's only because there is considerable commercial pressure and so we're just closing our eyes just because it's convenient. It may not last forever. (Or if it does, that means we're definitely even more stupid and greedy than I thought.)

[emece67]

.

[jfiresto]

Python might be slow, but while people who never actually used it write these highly opinionated long posts, that it's bad because -insert reason here-, using the same time we just write scripts that gets the job done....

I admit I was a little slow getting around to using Python (to rescue some colleagues' embedded system project in 2007 that would have failed had I not chosen Python). I would, however, say that what you write has been true since the mid-1990s. A related issue is Python can be so accommodating that many never learn to use it as a high level language.

EDIT: changed "Another issue ..." to "A related issue ..."

[voltsandjolts]

if the trend shifts back toward more efficient programming, it will be for energy efficiency reasons

If Python fails to meet the "runtime performance or efficiency" requirements, write those parts in C.
If C fails to meet the "development time performance or efficiency" requirements, write those parts in Python.

Netflix has been using Python for years but my guess is they don't using for the streaming code. In fact, they have an eye on alternatives to come in the near future.

But right now, it's horses for courses, where a combination of Thoroughbreds and Clydesdales makes a powerful team.

[Ed.Kloonk]

Watched a good video today about the Linux package manager, Nala.

It's a python wrapper for APT but it presents the information much cleaner. A perfect application for python. I did chuckle watching when the guy had to apt install 5000+ python depends along with the program.

[brucehoult]

if the trend shifts back toward more efficient programming, it will be for energy efficiency reasons

If Python fails to meet the "runtime performance or efficiency" requirements, write those parts in C.
If C fails to meet the "development time performance or efficiency" requirements, write those parts in Python.

It is notoriously hard to know in advance exactly which parts of a non-trivial application will be the actual bottleneck. If you guess wrong then you have to re-write Python code into C, or you spent time writing something in C initially when you didn't have to.

And the Python:C interface is not zero-friction, it has to be designed, so moving it up or down the software stack later has quite some cost.

I prefer to develop in a single language that I can write just as quickly as Python, but that runs much faster by default, and within a very small percentage of the speed of C with a little incremental work such as adding explicit type declarations or turning selected low level functions (or parts of a call tree) into macro / inline-only / template. WITHOUT extensive re-write to do it.

For the past 20 years my language of choice for this has been Dylan, a language designed by Apple in the early 90s to replace all their use of C++, SmallTalk, and Lisp.  Before that, the best option was Common Lisp (and in fact that is more viable than ever now). The modern "new" language implementing the same ideas (mostly) is Julia, although Swift is gradually getting closer and closer too.

[jfiresto]

Dylan looks very interesting but it is not even among TIOBE's top 100 languages.

I generally find Python is "low-friction" but once in a while, you have to play with a bit to discover a simple solution, probably because it has developed organically over many years to address many common problems. Sometimes your problem is not so common.

Parts of Python are showing signs of senescence. I needed to create a custom importer to tweak some DSL code so that it will compile. I looked at using importlib, but it appears to still be a work in progress. As of Python 3.10, it has over 7000 words of documentation, 12 supporting PEPs and 20 deprecated objects.

[brucehoult]

Dylan looks very interesting but it is not even among TIOBE's top 100 languages.

It's somewhat close to death :-(

In the mid to late 90s there were three major implementations:

1) Apple for MacOS. They went so far as announcing at WWDC that they were replacing C++ with Dylan throughout their code, and handing out CDs with a beta version of an IDE and compiler. Sadly, Dylan was one of the first things to get "Steve'd" when Apple was (for other reasons) losing billions of dollars per quarter and couldn't afford long term projects, no matter how promising.

2) CMU for Unix, specifically HP-UX. (PA-RISC "Snakes" was the best CPU around at the time)

3) Harlequin, a commercial vendor of Lisp, Smalltalk, Standard ML (and Haskell?), for Windows.

By 1998 the CMU people had graduated and moved on to other things. Fortunately they officially open-sourced (MIT license) what they had done, and I became part of a group (Dylan Hackers) that picked up and continued their work. Their compiler, "d2c", is as the name suggests a compiler that outputs very ASM-like C code. We very quickly ported it to work on x86 and PowerPC Linux, and to MacOS. It's very portable, I've used it myself on SPARC and MIPS and RISC-V.

d2c is still my favourite, because it is easy to interface to C (it generates C!), and its object representation doesn't steal any bits from machine pointers or integers, or float/double.

In 1999 Harlequin was bought by Global Graphics, who wanted them only for their PostScript interpreter. They shut down or spun off everything else. The Harlequin Dylan team managed to get the rights to the Windows IDE and compiler and libraries which, at the time, had something like 100 person-years of work in it (30 years in the "MPS" memory-manager / garbage collector alone). They formed a company, Functional Objects, and released Functional Developer (for Windows) in 2000. At first they were trying to sell it, and then later made it free and tried to make money with consulting and support. Eventually they simply turned all the code and rights over to the Dylan Hackers open source project, where it became the main focus of work.

Activity has trailed off, and I stopped being heavily involved when I switched my attention to RISC-V. But there continue to be releases with the most recent one being at the end of 2020. OpenDylan (nee Harlequin Dylan, nee Functional Developer) has gained an LLVM back end instead of x86 Windows native code generator, and from LLVM you can output webasm if you want.

You can actually play with Dylan in your web browser: https://play.opendylan.org/

Try the following:

Code: [Select]

define method fib (n :: <integer>)
  if (n < 2) n else fib(n - 1) + fib(n - 2) end
end;

format-out("%d", fib(39));

For me that takes 1.47 seconds to compile and run and output 63245986.  (I usually use fib(40) for benchmarks, but there seems to be a maximum CPU usage on the web server)

The equivalent Python takes 15.8 seconds on my M1 Mac. In C compile and run takes 0.53 with -O0 or 0.40 with -O2.

I don't actually know what server the Dylan playground is running on, but clearly it's closer to C than to Python.

You can make the Dylan code equivalent to Python by removing the type declaration on the function argument.

Code: [Select]

define method fib (n)
  if (n < 2) n else fib(n - 1) + fib(n - 2) end
end;

This makes it run more slowly. Sadly, the Playground time limit makes me drop it to fib(37), which prints 24157817 in 1.3 seconds. Python on my M1 Mac uses 6.4 seconds for that, C 0.26 (compile AND run).

A local install of OpenDylan will of course run faster and not have the CPU time limit!

Dylan, alone of these languages (but like Common Lisp or Julia), lets you write this function in a completely different way:

Code: [Select]

define method fib (n == 0) 0 end;
define method fib (n == 1) 1 end;

define method fib (n)
  fib(n - 1) + fib(n - 2)
end;

The online playground runs this for fib(37) in 1.3 seconds -- no detectable speed difference at all for the abstraction.

But now Dylan has another trick!

Code: [Select]

define open generic fib(n);
define method fib (n == 0) 0 end;
define method fib (n == 1) 1 end;

define method fib (n)
  let res = fib(n - 1) + fib(n - 2);
  add-method(fib, method(n == n) res end);
  res
end;

format-out("%d", fib(37));

This still outputs fib(37) = 24157817 in 1.3 seconds.

BUT ... now we can do fib(89) = 1779979416004714189 in still 1.3 seconds.

Try that with the recursive C or Python code and you'll be there a very very long time. I've tried to estimate, and I think the C code (with -O2) would take about 350 years.

[olkipukki]

It is notoriously hard to know in advance exactly which parts of a non-trivial application will be the actual bottleneck. If you guess wrong then you have to re-write Python code into C, or you spent time writing something in C initially when you didn't have to.

It's not black and white, there is a grey area, but closer to Python rather than C

[brucehoult]

cython got you a speedup of 4.3x.

C gets you *another* factor of 10 to 20 speedup

As demonstrated in my post immediately before yours, Dylan code looks like Python (and is as fast/easy to write), but runs much much closer to C speed than cython does.

On my x86 Linux machine (1st gen ThreadRipper):


24.2  Python 3.9
 5.1  Dylan with no declarations, base cases handled by virtual function dispatch (separate method declarations)
 4.8  Dylan with no declarations (same style as Python)
 1.3  Dylan with fib argument declared as <integer>
 1.0  Javascript with no declarations (see below)
 0.75 C without -O
 0.7  Dylan with fib return value also declared as <integer>
 0.5  C with -O1
 0.2  C with -O2


The JavaScript source code:

Code: [Select]

function fib(n) {
  return n < 2 ? n : fib(n-1) + fib(n-2);
}

console.log(fib(40));

Running the Javascript:

Code: [Select]

$ time node fib.js
102334155

real 0m1.034s
user 0m1.031s
sys 0m0.004s


[Picuino]

PHP, Matlab and other interpreted languages would be missing in the comparison. The obvious conclusion is that interpreted languages should not be used because, in all cases, they can be replaced by much faster compiled languages.

The reality is that, surprisingly, interpreted languages are still used extensively.

[Picuino]

In the field of teaching/learning, interpreted languages have the advantage of being much faster in the trial and error cycle that is so didactic when learning a new language or when learning to program.
In the case of block programming, Scratch is the most popular. Based on Javascript, it has been ranked 35th on Tiobe for several months. Not bad for a language aimed at children learning programming. But in the case of text languages, I prefer Python to Arduino's C++.

[brucehoult]

In the field of teaching/learning, interpreted languages have the advantage of being much faster in the trial and error cycle that is so didactic when learning a new language or when learning to program.

That hasn't been true for student-sized programs written in C/C++. Pascal or similar since ... I was going to say since Linux on an original Pentium, but really, it's since Turbo Pascal 1.0 on a Z80. Or THINK Pascal / THINK C on a Mac Plus.

[tellurium]

This still outputs fib(37) = 24157817 in 1.3 seconds.

BUT ... now we can do fib(89) = 1779979416004714189 in still 1.3 seconds.

What?? How does that work.. Staring at the code and can't figure it out.

[brucehoult]

This still outputs fib(37) = 24157817 in 1.3 seconds.

BUT ... now we can do fib(89) = 1779979416004714189 in still 1.3 seconds.

What?? How does that work.. Staring at the code and can't figure it out.

Ah, someone didn't just skip over...  :-)

It dynamically adds new methods to the fib() generic function (virtual function in C++ terms).

It's like doing this in Python, except what is being added to the dictionary are functions (closures) not just the answer:

Code: [Select]

fibs = {}

def fib(n):
    if n in fibs:
        return fibs[n]
    if n in {0, 1}:
        return n
    res = fib(n-1) + fib(n-2)
    fibs[n] = res
    return res

print (fib(89))


[SiliconWizard]

In the field of teaching/learning, interpreted languages have the advantage of being much faster in the trial and error cycle that is so didactic when learning a new language or when learning to program.

That hasn't been true for student-sized programs written in C/C++. Pascal or similar since ... I was going to say since Linux on an original Pentium, but really, it's since Turbo Pascal 1.0 on a Z80. Or THINK Pascal / THINK C on a Mac Plus.

Yep.

Now, the interactive nature of interpreted languages still makes them... not necessarily "faster" for development, but more... interactive. Yeah.
So if you're constantly "trying" things, cutting the modify/save/compile cycle, even if it's pretty short, seems more convenient for some - in particular beginners.

Thing is, it does promote a pretty "tinkery" approach to programming, which is definitely not how one should properly *learn* some new skill IMHO, at least if it's in any serious way.
That's pretty much what happened with BASIC. We know how that goes.
It's "learning" as if you were a fly, just flying around your target in all directions until you finally hit your goal and move on.

But it's precisely the strength of languages such as Python: they appeal to non-programmers, which opens a gigantic "market". So many of its users are precisely people who do *not* want to learn programming in a sane way. They just want to get some results - even if they don't quite grasp how they got there, who cares, if they get stuck they'll just flood online forums.

[tellurium]

It's like doing this in Python, except what is being added to the dictionary are functions (closures) not just the answer:

Ah, got it! That's clever..
For that particular function, it's possible to memoize results, rather than methods, and get a boost:

Code: [Select]

$ cc -O2 ~/tmp/fib.c -o /tmp/fib && /tmp/fib 45
recursive, result: 1134903170, time: 3.408709 sec
memoised , result: 1134903170, time: 0.000003 sec

Source:

Code: [Select]

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

static unsigned long fib(unsigned long n) {
  return n < 2 ? n : fib(n - 1) + fib(n - 2);
}

static unsigned long fib2(unsigned long n) {
  static unsigned long tab[100];
  if (tab[n]) return tab[n];
  tab[n] = fib(n);
  return tab[n];
}

static void measure_time(const char *msg, unsigned long (*fn)(unsigned long),
                         unsigned long n) {
  clock_t start = clock();
  unsigned long result = fn(n);
  clock_t duration = clock() - start;
  printf("%s, result: %lu, time: %.6f sec\n", msg, result,
         (double) duration / CLOCKS_PER_SEC);
}

int main(int argc, char *argv[]) {
  unsigned long n = 1;
  if (argc > 1) n = strtoul(argv[1], NULL, 0);
  measure_time("recursive", fib, n);
  measure_time("memoised ", fib2, n);
  return 0;
}


[brucehoult]

It's like doing this in Python, except what is being added to the dictionary are functions (closures) not just the answer:

Ah, got it! That's clever..
For that particular function, it's possible to memoize results, rather than methods, and get a boost:

Right, except the code you present can't possibly work.  fib2() needs to recursively call itself, not fib().

In Dylan you get full virtual function dispatch on the newly-added methods, and in a way that is more powerful than in C++. You can dispatch based on the most specific subclass (as in C+, Java, Python etc), or on various derived types such as singleton() (as seen in my code there), ranges of values, unions of different types. AND the virtual dispatch is done using ALL arguments of the method, not just the first one.

[ve7xen]

The question is what would make a good introductory language now? It certainly isn't assembler-lite like CESIL was, but neither is it a language like Python that throws in everything including the kitchen sink and has idiosyncrasies that make it non-representative of programming languages as a whole.

I'm really not convinced that this idea that Python is fundamentally a bad language for introducing Computer Science holds because it 'throws in everything'. The argument seems to be that 1. rigour and 2. lack of alternatives contributes to a language's success as a teaching language. In other words, a language that forces your hand into doing things 'properly', because there simply is no other way. I'm not sure this is actually good for learning the underlying 'whys', though, and actually understanding the fundamental concepts that a computer science education is trying to give you.

What it also means, more practically, is that such a language will have a narrow scope - it might be suitable for object-oriented programming, but not functional programming, and so on, because that would offer too many choices and the flexibility to do stupid things that don't make sense. This leads to a lot of time and energy wasted on learning the syntax and boilerplate associated with the 'appropriate' language for the given topic, rather than the actual computer science you're there to learn. Instead of computer science, you spend a lot of time learning a handful of languages poorly, and precious little learning concepts (ie. 'programming' and not 'computer science'). In fact, you make this point yourself ('too many introductory courses to programming are introductory courses to programming in language x'), so I am surprised that you don't see that the alternative to this is a flexible language like Python that includes the constructs necessary for any sort of programming.

I believe that such an education, like any other, should be guided by a teacher, who can introduce the computer science concepts in a natural progressive fashion, while providing feedback and guidance based on the students' choices. Python lends itself well to this approach, as it is flexible enough to support almost any computer science concept, and with a simple and easy to learn syntax that can carry a student from a grade-school one-line Hello World through the vast majority of computer science outside of systems programming. Sure 'Pythonic' code is fairly idiosyncratic due to some 'unique' language features - like idiomatic code in any language - but you don't have to teach that style (and the reasoning behind it). At least until the fundamentals are well understood, at which point it becomes a marketable skill in one of the most popular languages currently in use, and not just a 'toy' you used to learn like most teaching languages have been or likely will be.

I'm sure Python is not the best choice to teach each branch of computer science, but it is fairly decent for most of them, and certainly for important fundamentals like data structures, OO, functional etc, and is actually useful and used in the real-world too. And as someone who had to struggle to stay awake through the first half of courses being 'learn basic Java' or 'learn basic Scheme' or 'learn basic Perl' before I could actually get anything useful out of them, I think it would be a vast improvement on the way I was taught 15 years ago to have a nice linear progression through these concepts using the same language with the same syntax that doesn't burn half the course's instruction time and a lot of student effort in solving basic syntax problems that are decided not computer science related.

They actually succeeded in teaching 6 year olds to program using it. Remember that's about when you start to learn to read and write the language that you've grown up with, language that will be parsed with a whole human brain for syntax and semantics, not the pathetic fussy brain of a literal computer. It's quite an achievement to have got six year old to write any code that actually works.

I don't know about you, but I find simple Smalltalk quite a bit more inscrutable than Python, so I'd wager you can teach those same 6-year-olds basic Python too. However, I don't think 6-year-olds are a good judge of a language for teaching computer science, which is about high-level conceptual ideas and not making the computer do something. To get a reasonably good sense of what programming is about, before formal computer science, Scratch is much better suited, and completely obviates the need for 'syntax' as we know it.