Python becomes the most popular language

[brucehoult]

According to Pareto's law, 80% of the code only takes 20% of the execution time, while 20% of the code takes 80% of the execution time.
Normally you should only optimize 20% of the code to gain a lot of speed.

That's true if you write all the code in the same language (or at least similarly efficient languages).

In Python this 20% of the code can already be optimized with external libraries, sometimes written in C (like numpy).

Let's try a thought experiment.

We have some task written in C (or Java or C#) that takes 10 seconds to run:

8 seconds: the important 20% of the code
2 seconds: the "unimportant" 80% of the code that you want to be easy to maintain

We rewrite 80% of the code in Python. Python runs 50 times slower than C. Now we have:

8 seconds: the important 20% of the code (still in C)
100 seconds: the Python part

Total runtime just went from 10 seconds to 108 seconds. The Python code is taking 92.6% of the time.

I sure hope it's a LOT easier to maintain to make it worth it.

Or, it had better be more like 99%/1% in the C version, not 80%/20%.


(Of course 108 seconds is better than the 500 seconds it would take if you wrote it all in Python ... but 10 seconds is even better)

You're grossly over estimating speed differences.

No, that's accurate. If you write the same algorithm the same way in C and Python then the Python is about 50 times slower. That's well-established e.g. see



People argue he's not using idiomatic Python and THAT'S MY POINT. Python has a huge penalty for Just Writing Code instead of using the built-in high level abstractions (that were written by someone else, in C).

There's plenty of examples floating around of people writing some numerical routine in pure C/C++ and it being slower than numpy because said person doesn't know how to write the code to squeeze every bit of optimization from the compiler.

That just means the person writing the C/C++ doesn't know as much about numerical programming as the person who wrote numpy. Numpy can't be faster than C because numpy *is* C. Using an appropriate algorithm and coding is of course important, and if you're not competent to write that yourself then you should use a library.

Numerical libraries are also available in C. NAG, LINPAK, BLAS etc have been around for many decades. Hell -- you can use numpy from a C program too, if you want to.

People using numpy are probably somewhere around 99.99% of the executed operations are in numpy, 0.01% in Python. That's very far from 80/20.

Edit: I don't even understand what you are arguing here. Its like you're completely ignoring that fact that for super hot spots in the code you can write C/C++ and use it directly in the python code.

How can I be ignoring it, when that's exactly the whole thing I'm talking about?

[jfiresto]

... People don't write python for speed, they write python for the user interface experience. The speed comes from writing stuff in C/C++....

I have a question. I am never short of problems that need solving. I write in Python, and other high level languages, to solve a problem more thoroughly and faster, so that I can move on to the next one. Is that what you mean by the user interface experience?

[mansaxel]

... People don't write python for speed, they write python for the user interface experience. The speed comes from writing stuff in C/C++....

I have a question. I am never short of problems that need solving. I write in Python, and other high level languages, to solve a problem more thoroughly and faster, so that I can move on to the next one. Is that what you mean by the user interface experience?

It simply means that for your workload, the compute time is small compared to the problem formulation time, making time-to-solution more dependent on "write fast" than "compute fast".

With the speed of modern computers taken into consideration (anything built the last 20 years excluding microcontrollers) this is the situation for nearly all of the problems people experience in their professional lives. The exceptions are few, and include things like high energy physics, fluid dynamics, some graphics work, and of course the utterly insane practice of proof-of-work pyramid schemes.

My impression is that most CPU time these days is spent waiting, computing things to be drawn on screen, and drawing them. This means that interpreted languages are not nearly enough that much of a problem.  I usually write in awk because what I do tends to center around line-oriented text files that need transformation and some math. Also interpreted, but without the completely opaque jungle of add-ons that are curl|sudo bash -installed.

[jfiresto]

Thank you for the explanation. I did guess your gist.

[Just_another_Dave]

According to Pareto's law, 80% of the code only takes 20% of the execution time, while 20% of the code takes 80% of the execution time.
Normally you should only optimize 20% of the code to gain a lot of speed.

That's true if you write all the code in the same language (or at least similarly efficient languages).

In Python this 20% of the code can already be optimized with external libraries, sometimes written in C (like numpy).

Let's try a thought experiment.

We have some task written in C (or Java or C#) that takes 10 seconds to run:

8 seconds: the important 20% of the code
2 seconds: the "unimportant" 80% of the code that you want to be easy to maintain

We rewrite 80% of the code in Python. Python runs 50 times slower than C. Now we have:

8 seconds: the important 20% of the code (still in C)
100 seconds: the Python part

Total runtime just went from 10 seconds to 108 seconds. The Python code is taking 92.6% of the time.

I sure hope it's a LOT easier to maintain to make it worth it.

Or, it had better be more like 99%/1% in the C version, not 80%/20%.


(Of course 108 seconds is better than the 500 seconds it would take if you wrote it all in Python ... but 10 seconds is even better)

You're grossly over estimating speed differences.

No, that's accurate. If you write the same algorithm the same way in C and Python then the Python is about 50 times slower. That's well-established e.g. see



People argue he's not using idiomatic Python and THAT'S MY POINT. Python has a huge penalty for Just Writing Code instead of using the built-in high level abstractions (that were written by someone else, in C).

There's plenty of examples floating around of people writing some numerical routine in pure C/C++ and it being slower than numpy because said person doesn't know how to write the code to squeeze every bit of optimization from the compiler.

That just means the person writing the C/C++ doesn't know as much about numerical programming as the person who wrote numpy. Numpy can't be faster than C because numpy *is* C. Using an appropriate algorithm and coding is of course important, and if you're not competent to write that yourself then you should use a library.

Numerical libraries are also available in C. NAG, LINPAK, BLAS etc have been around for many decades. Hell -- you can use numpy from a C program too, if you want to.

People using numpy are probably somewhere around 99.99% of the executed operations are in numpy, 0.01% in Python. That's very far from 80/20.

Edit: I don't even understand what you are arguing here. Its like you're completely ignoring that fact that for super hot spots in the code you can write C/C++ and use it directly in the python code.

How can I be ignoring it, when that's exactly the whole thing I'm talking about?

The same thing could happen in the other direction. Copying an idiomatic python algorithm to C might result in inefficient code (that’s one of the problems of Nuitka). Speed comparison should be done with the most efficient implementation of the algorithm for each language (not different algorithms) to provide useful results. A good example of this is the Ada implementation, as one of its inefficiencies is a consequence of the chosen types for the variables (due to the number of operations required to read and write them) instead of the actual algorithm

[Picuino]

...
Python runs 50 times slower than C.
...

Not that much. two to ten times slower.

Edit:
It depends on the type of program you are doing.
But even with a 50 to 1 difference, it is still better in many cases to program in Python. That is why it is programmed in Python, not because programmers do not know how to program in C!

I am trying to find out why Python is so popular. If you show me that there are no reasons for Python to be popular, that is against reality!

[Picuino]

Why is Google using Python on its Youtube service?
Is Google Against Speed? I do not believe it.
Doesn't Google know the advantages of C? I do not believe it

The question remains why Python has become such a popular language, the first one.

[brucehoult]

...
Python runs 50 times slower than C.
...

Not that much. two to ten times slower.

Edit:
It depends on the type of program you are doing.
But even with a 50 to 1 difference, it is still better in many cases to program in Python. That is why it is programmed in Python, not because programmers do not know how to program in C!

It can be much more than that. In the video referenced above, it was found that running that same algorithm, counting the number of primes below one million, the number of iterations done in five seconds was:

  25: Python
2651: C# (106x faster)
7436: C++ (297x faster)

[cfbsoftware]

I am trying to find out why Python is so popular.

I haven't read it myself but I suspect there will be a number of clues and possible answers to your question in this book:

https://www.amazon.com.au/Hit-Makers-Science-Popularity-Distraction/dp/110198032X

Alternatively, try this article:

https://betterexplained.com/articles/logical-fallacy-popularity-is-not-quality/

[xrunner]

Why is Google using Python on its Youtube service?

I do not know. Did you attempt to ask Google why? What did they tell you?

Is Google Against Speed? I do not believe it.

I do not know. Did you attempt to ask Google? What was the answer?

Doesn't Google know the advantages of C? I do not believe it

I do not know. Did you attempt to ask Google? What was the answer? You don't have answers from Google but you certainly do tell us what you believe. And I do believe that you believe it.

The question remains why Python has become such a popular language, the first one.

OK so we have a question that remains, which means we don't have the answer.

I already pointed out to you here -

https://www.eevblog.com/forum/programming/python-becomes-the-most-popular-language/msg3732397/#msg3732397

The PYPL PopularitY of Programming Language Index is created by analyzing how often language tutorials are searched on Google.

The more a language tutorial is searched, the more popular the language is assumed to be. It is a leading indicator. The raw data comes from Google Trends.

https://pypl.github.io/PYPL.html

Also from something recently referenced in this thread but now is gone (did someone edit their post?) -

https://www.tiobe.com/tiobe-index/

It is important to note that the TIOBE index is not about the best programming language or the language in which most lines of code have been written.

What is the definition of "popular"?

1.
liked, admired, or enjoyed by many people or by a particular person or group.
"she was one of the most popular girls in the school"

2.
(of cultural activities or products) intended for or suited to the taste, understanding, or means of the general public rather than specialists or intellectuals.
"the popular press"

definition from Oxford languages/Google

You need to be very careful when drawing conclusions from something that appears to be popular to something that is better.  That's what you are attempting to do, go from "it's popular" to "it's better". For example, if tutorials for something are searched more for thing A than for thing B, does that mean thing A is better? No. It might be, but what are some other reasons that might happen? It might be that thing A is harder to learn, which is why the tutorials are being searched more than the other thing. It might be that thing A is much newer and people want to learn it, meanwhile thing B has been around for a long time and people already understand it, so less tutorials are searched. I can think of other reasons as well - none of which have anything to do with thing "A" being "better" than thing "B"

I wouldn't even go so far as saying a thing that has the most searches for tutorials is popular, looking at the definition. It might be that a lot of students hate the thing, but they are required to learn it because it's being taught more these days. Why might it be taught more these days? That's a question, and I don't know the answer (if that is true). I can speculate as well as anyone here, but it wouldn't mean squat.

Note: I do not have any dislike for any particular programming language, but I do dislike drawing wrong conclusions.

[Just_another_Dave]

I am trying to find out why Python is so popular. If you show me that there are no reasons for Python to be popular, that is against reality!

From my point of view, it is a combination of being a language designed to teach beginners (it is a batteries included language with dynamic types that handles many complexities for you) and a similar sintaxis to matlab with a good integration with c libraries (that made it appealing to programmers that wanted a free substitute to matlab). In addition to that, its module system and pip make using third party libraries as easy as using the C standard library, which is a great advantage. As a result, it is appealing for beginners because it is used for building real applications in fields with a lot of hype like machine learning, which was the problem of other beginner friendly languages like basic (it was seen as a toy language, despite being used for building many internal programs due to its lack of use in fashionable applications)

Being popular also means that there are many forums where people can ask questions and that many programs offer a python api for automating their use

[SiliconWizard]

People argue he's not using idiomatic Python and THAT'S MY POINT. Python has a huge penalty for Just Writing Code instead of using the built-in high level abstractions (that were written by someone else, in C).

Yes, and that point actually applies to much more than just speed.

Python as a programming language is rather questionable - to say the least. Of course that would  take quite a while to get into details, and some may not agree.

But point is, at least from the majority of Python uses I've seen - so admittedly that's partly subjective - people using Python usually do it not particularly for the merits of the language itself, but for the enormous pile of existing libraries that you can find for about any purpose. And this can be perfectly seen in this thread as well.

Python is not the only language that became popular thanks to available "libraries" rather than the core language itself, by the way. This is one of the most common way of making a language popular. (Of course there must always be a trigger - libraries do not appear all at once by magic - but this is sort of a feedback loop).

[jfiresto]

... Python as a programming language is rather questionable - to say the least. Of course that would  take quite a while to get into details, and some may not agree....

Yes, some may and some may not agree. Out of curiosity, and nothing more than that, do you prefer C++?

[Nominal Animal]

People argue he's not using idiomatic Python and THAT'S MY POINT.

Also my own perennial issue with programmers that expect their experience and knowledge in one programming language and environment to be directly applicable in another.  (Instead of "idiomatic language", I just call it the paradigm.)

The good thing about high-level languages like Python, Ruby, and even awk, is that instead of the basic imperative operations (loops and conditionals), you can directly skip into event-based programming and algorithms.  For example, dictionaries (noting that in awk, all arrays are dictionaries) provide an efficient tool for generating graphs and disjoint sets.  Then, describing these as abstractions involving quite interesting lower-level code implementing them, showing that by choosing the tool used, the programmer can choose at which level of complexity to work on the problem, and what the tradeoffs are, the programmers hopefully learn that a single programming language is not 'best', and that each programming language has their paradigm, their own idiomatic style, that lets one solve problems most efficiently using that language.

Numerical libraries are also available in C. NAG, LINPAK, BLAS etc have been around for many decades. Hell -- you can use numpy from a C program too, if you want to.

People using numpy are probably somewhere around 99.99% of the executed operations are in numpy, 0.01% in Python. That's very far from 80/20.

Exactly.

Until recently, I experimented a lot with a linear algebra library interface that I hoped would let scientists write the code in C instead of Python, Matlab, Octave, and such tools.  (The reason why in C, is complicated and boils down to running distributed code in clusters, using MPI to communicate between instances.  However, in the past few years, I believe mpi4py (packaged in Debian-derivatives as python3-mpi4py and in RHEL-derivatives as python3-mpi4py-mpich or python3-mpi4py-openmpi) has become quite acceptable, as outlined in this ieee article.)

Basically, both matrices and views to matrices were indistinguishable, as data was stored in separate refcounted (or GC'd) structures, allowing one to have vectors and even transposes to the same matrix without issues in single-treaded code.  It works, and is a nice C interface.  However, it really gains nothing compared to using idiomatic Numpy/Scipy or Matlab/Octave stuff.  All it does is make the linear algebra code implemented in C a bit simpler/cleaner compared to GSL, Intel core math library, AMD core math library, BLAS, LAPACK, et cetera.

Right now, for a scientist doing numerical calculations using NumPy/SciPy, I see no reason why they could not do distributed processing in a cluster using mpi4py as well, exactly because 1) the CPU time used to run Python is a tiny fraction of the whole, and 2) it is not difficult to teach/learn the patterns and algorithms that lead to efficient distributed code.  Of course, without such guidance, the Python code they generate is likely completely crap; but so is self-taught C, in most cases.

Looking back, this seems so obvious now.

[tszaboo]

People argue he's not using idiomatic Python and THAT'S MY POINT. Python has a huge penalty for Just Writing Code instead of using the built-in high level abstractions (that were written by someone else, in C).

Maybe they argue, because that's how you use python. If you want a chart, you use matplotlib. If you want data science, you import pandas. If you want to calculate something fast, you import numpy, cuda or ray or opencv.
I dont get your arguments. Nobody is going to write code the way it is in that Youtube video, or how you are saying, because it is slower to write and slower to run. And counterintuitive.
And any speed argument for pure computational script is kinda useless. If I need to calculate something fast in python, I can always import ray, change maybe 3 lines in the code, and then run it distributed on a server park. Or import any of the libraries that make it run on video cards, 1000x faster than your best C implementation. It can utilize power you can never dream of, just because doing the same thing in C is hilariously complicated.

[mansaxel]

The good thing about high-level languages like Python, Ruby, and even awk, is that instead of the basic imperative operations (loops and conditionals), you can directly skip into event-based programming and algorithms.  For example, dictionaries (noting that in awk, all arrays are dictionaries) provide an efficient tool for generating graphs and disjoint sets.  Then, describing these as abstractions involving quite interesting lower-level code implementing them, showing that by choosing the tool used, the programmer can choose at which level of complexity to work on the problem, and what the tradeoffs are, the programmers hopefully learn that a single programming language is not 'best', and that each programming language has their paradigm, their own idiomatic style, that lets one solve problems most efficiently using that language.

But there is great fun to be had in playing with languages and using them to solve any problem, especially so classes of problems that are decidedly unsuited to that particular language, as most often used.

Right now, for a scientist doing numerical calculations using NumPy/SciPy, I see no reason why they could not do distributed processing in a cluster using mpi4py as well, exactly because 1) the CPU time used to run Python is a tiny fraction of the whole, and 2) it is not difficult to teach/learn the patterns and algorithms that lead to efficient distributed code.  Of course, without such guidance, the Python code they generate is likely completely crap; but so is self-taught C, in most cases.

Looking back, this seems so obvious now.

I once worked at a computing centre where one of the compute clustres essentially was a speed test of Perl and the AFS client...

[Nominal Animal]

But there is great fun to be had in playing with languages and using them to solve any problem, especially so classes of problems that are decidedly unsuited to that particular language, as most often used.

Well, having fun is having fun, and as long as nobody gets harmed (say, get hoisted to maintain the result because someone found it and decided to actually use it and then left for greener pastures), it is just fun.

Sometimes, I check out some of the submissions to IOCCC, just for fun.  But if when I see that kind of code in real life, it makes me die a little inside.

I once worked at a computing centre where one of the compute clustres essentially was a speed test of Perl and the AFS client...

CERN still uses a lot of Java, including in distributed computing.  Compared to running Perl, that's an ouchie, in my opinion.

[Picuino]

Comparing speed of C and Python with equal algoritm:

Code: (Python) [Select]

import time

def eratostenes(n):
    nums = [1 for i in range(n+1)]
    nums[0] = nums[1] = 0
    for i in range(2, n+1):
        if nums[i]:
            x = i * i
            if x > n:
                break
            for j in range(x, n+1, i):
                nums[j] = 0
    return nums

def main():
   timeit = time.time()
   for i in range(1000):
       nums = eratostenes(1000000)
   timeit = time.time() - timeit
   print('time =', timeit)

main()
130 Seconds

Code: (C) [Select]

#include <stdlib.h>
#include <stdio.h>
#include <time.h>

char *eratosthenes(unsigned long n) {
   char *nums, *p;
   unsigned long i, x;

   /* Initialize */
   nums = malloc(n + 1);
   p = nums;
   *p++ = 0;
   *p++ = 0;
   for(i=2; i<=n; i++)
      *p++ = 1;

   /* Find primes */
   for(i=2; i<=n; i++) {
      if(nums[i] == 1) {
         x = i*i;
         if (x > n)
            break;
         while(x<=n) {
            nums[x] = 0;
            x += i;
         }
      }
   }
   return nums;
}

int main() {
   char *nums;
   int i;
   clock_t timeit;
   
   timeit = clock();
   for(i=0; i<1000; i++) {
      nums = eratosthenes(1000000);
      free(nums);
   }
   timeit = clock() - timeit;
   printf("time = %f\n", (float)(timeit) / CLOCKS_PER_SEC);
}
2.36 Seconds

Relationship = 130/2.36 = 55 : 1

[Ed.Kloonk]

You know.. most uses for Python is not just sitting there crunching primes. I remember Video games used to be written in C and the tricky bits were in ASM.

How hard is it to shim in a bit 'o C when the going gets tough? Srs. Q.

[Marco]

Most code in gaming is incorporating user input and updating gamestate at some glacial pace with the engine handling rendering, pathfinding and physics. Pathing conflicts and collisions simply handled non-iteratively on the next tick. Game logic can generally be done in a script language no problem, computers keep getting faster but 60 Hz stays the same.

Mostly C# and Unrealscript though, not Python.

[Nominal Animal]

But point is, at least from the majority of Python uses I've seen - so admittedly that's partly subjective - people using Python usually do it not particularly for the merits of the language itself, but for the enormous pile of existing libraries that you can find for about any purpose. And this can be perfectly seen in this thread as well.

True; I just don't think it is a serious drawback, because it is a tool that works for such purposes.

In fact, I consider it a serious positive compared to say Ruby and Lua; especially because it is easy to do in Python for my own libraries, including those not specially designed for any kind of Python interfacing.

Python is not the only language that became popular thanks to available "libraries" rather than the core language itself, by the way. This is one of the most common way of making a language popular. (Of course there must always be a trigger - libraries do not appear all at once by magic - but this is sort of a feedback loop).

For Python, the trick is that you don't need to do any kind of changes to the libraries; the interfacing only requires Python code, which really is revolutionary among programming languages.

It is kinda funny that such a technical detail, which has almost nothing to do with the language itself, has been such crucial factor in its success.

[Picuino]

Quicksort algorithm over long long numbers:

Code: (Python) [Select]

import random
import time

def partition(values, left, right, pivotidx):
    pivot = values[pivotidx]
    values[right], values[pivotidx] = values[pivotidx], values[right]
    storeidx = left
    for idx in range(left, right):
        if values[idx] < pivot:
            values[idx], values[storeidx] = values[storeidx], values[idx]
            storeidx += 1
   
    values[storeidx], values[right] = values[right], values[storeidx]
    return storeidx

def _doquicksort(values, left, right):
    if right > left:
        pivotidx = (left+right)//2
        pivotidx = partition(values, left, right, pivotidx)
        _doquicksort(values, left, pivotidx)
        _doquicksort(values, pivotidx + 1, right)
    return values

def quicksort(nums):
    return _doquicksort(nums, 0, len(nums) - 1)
       
def main():
   nums = [random.getrandbits(64) for i in range(1000000)]
   timeit = time.time()
   nums = quicksort(nums)
   timeit = time.time() - timeit
   print('time =', timeit)

main()

Code: (c) [Select]

#include <stdlib.h>
#include <stdio.h>
#include <time.h>

#define ARRAYSIZE   1000000

long partition(long long *values, long left, long right, long pivotidx) {
   long long temp, pivot;
   long storeidx, idx;
   
   pivot = values[pivotidx];
   temp = values[right];
   values[right] = values[pivotidx];
   values[pivotidx] = temp;
   storeidx = left;
   for(idx = left; idx<right; idx++) {
      if (values[idx] < pivot) {
         temp = values[idx];
         values[idx] = values[storeidx];
         values[storeidx] = temp;
         storeidx++;
      }
   }
   
    temp = values[storeidx];
    values[storeidx] = values[right];
    values[right] = temp;
    return storeidx;
}

long long *_doquicksort(long long *values, long left, long right) {
   long pivotidx;

   if (right > left) {
      pivotidx = (left + right) / 2;
      pivotidx = partition(values, left, right, pivotidx);
      _doquicksort(values, left, pivotidx);
      _doquicksort(values, pivotidx + 1, right);
   }
   return values;
}

long long *quicksort(long long *nums) {
   return _doquicksort(nums, 0, ARRAYSIZE-1);
}
       
int main() {
   long i;
   clock_t timeit;
   long long *nums;
   
   
   nums = malloc(sizeof(long long)*(ARRAYSIZE+1));
   for(i=0; i<ARRAYSIZE; i++) {
      nums[i] = (long long)rand();
   }
   
   timeit = clock();
   nums = quicksort(nums);
   timeit = clock() - timeit;
   printf("time = %f\n", (float)(timeit) / CLOCKS_PER_SEC);
}
Python Time = 3.5579
C Time = 0.2170

Relationship = 3.5579 / 0.2170 = 16.4 : 1

[DiTBho]

Well, having fun is having fun

just for fun I tried to write a relativistic wave equation version of Erwin Schrödinger's equation (simplified), but gave up because it was too "serious" to be a hobby and I prefer to write some funny sentence to a Youtuber who talks about physics, until just for fun she pointed out a dude who actually made the point about the Dirac's particle physics equation.

• (my common operative field) For an electrical Engineer, electrons are irrelevant, what matters is current and voltage
• (what I made for fun) For Solid State Device Physic Engineer, current and voltage are not enough and electrons are not irrelevant, what matters is potential holes, and you use the Erwin Schrödinger's equation to describe them
• (what she replied for her personal fun) For an high-energy Physics, potential holes are not enough and you want to describe a particle that follows Fermi–Dirac statistics and generally has half odd integer spin, so you talk about spin, "fermion particles" as "excitations of a quantum field" (what?), and you use the Dirac's particle physics equation to describe them

So the point is what is fun?

An extraordinarily impressive increase in complexity that made the whole thing look like an episode of the television series "Big Bang Theory"

[tszaboo]

Comparing speed of C and Python with equal algoritm:

But the python code doesn't look like this.
It looks like this:

Code: [Select]

import timeit
timeit.timeit("list(sympy.sieve.primerange(0, 1000000))", setup="import sympy", number =1000)
Two lines of code, same algorithm, 8.23 seconds to run.
There are differences between the languages. One is that python will do garbage collection after every for cycle. Or you can write your code properly, and then it runs fast, and you actually didn't need to think about how to code,  just what to code.

[brucehoult]

Quicksort algorithm over long long numbers:

Python Time = 3.5579
C Time = 0.2170

Relationship = 3.5579 / 0.2170 = 16.4 : 1

I think you may be mostly measuring startup time there. Try with a bigger array, or do it multiple times.