Yet another icon stepping away - Guido van Rossum, Python

[Fixpoint]

I am suggesting that if a project grows to where the benefits of strong typing, variable localization and many other features of a language well suited to large projects it should bite the bullet and select an appropriate language.

You are completely right. But there are two problems:

• Those benefits you are talking about are needed pretty soon, much much sooner as most people think who are just used to tinkering and hacking. In a professional environment, software grows extremely quickly -- and yes, even that software that was never supposed to grow in any way! So, you need to lay the foundations for that right from the beginning. You cannot do that later. In a colloquial sense, software architecture can be described as those properties that cannot be changed but only re-engineered.
• Real-world contraints prohibit any kind of re-engineering. No boss, no client, and no deadline will ever ever ever allow me to scrap the existing code base and rebuild the system. It's not gonna happen. I have been working in the software industry for many years now, and I always advocate for solid designs. And if I know one thing then that a solid design must be built right from the start because refactorings are not gonna be approved. Even if, then only under very limited conditions. And it is not fun at all. Why do a refactoring if you could have done it right in the first place?

It would have been better to make that selection at the outset, but there are many reasons that doesn't happen.

EXACTLY.

Rather than upgrading the language, port the legacy code to a language of your choice.

Ok. But who pays for that? The client will give you the finger. Even for a small to midsized project, that can cost millions of dollars. It's not only simple arithmetics that has to be translated. Whole technologies must be exchanged because they are bound to a specific programming language. You might have to rebuild complete frameworks. If your software is safety-critical, the complete risk analysis and certification (expensive and time-consuming) has to be done again. And third-party software might have to be adjusted too! It can be an enormous, endless endeavor. It just doesn't work that way.

Think of it this way.  Would you like to see FORTRAN upgraded to modern programming standards, and if so, why?  I wouldn't.  But for its original purpose it remains one of the best languages around.

No, I don't want to see FORTRAN *anywhere*. I had to deal with it, and it's just an abomination. The code is a basically a time travel back into the earliest beginnings of human reason.

[tszaboo]

Probably the same things you do.  The kind of requirements that have changed for Python. 

The only argument for changing the implementation of the language rather than changing the language used for the task is the investment in code.  But that investment is thrown out the window if the new rules don't allow for backward compatibility.

I think you misunderstand me. I think python is an excellent language, starting from 4 liner BASH replacement to massive software protects. It is extremely versatile language. You can write your code on your laptop, and changing a few lines, you can run it on multiple CPU cores, multiple servers, Google TPU, your video card or heck, you can run massive computation projects on AWS.

Rather than upgrading the language, port the legacy code to a language of your choice.

Ok. But who pays for that? The client will give you the finger. Even for a small to midsized project, that can cost millions of dollars. It's not only simple arithmetics that has to be translated. Whole technologies must be exchanged because they are bound to a specific programming language. You might have to rebuild complete frameworks. If your software is safety-critical, the complete risk analysis and certification (expensive and time-consuming) has to be done again. And third-party software might have to be adjusted too! It can be an enormous, endless endeavor. It just doesn't work that way.

Actually, people do this, because the performance gain is immense. You could write maybe coprocessing FORTRAN,  but it is the second circle of hell. And then it runs on your CPU. If you have the same calculation done in python, you have to replace "import numpy" with "import cupy" ... and suddenly your code runs maybe 500 times faster, and the limit is just how much you are willing to pay for hardware or the time on someone else's computer.

[0culus]

Nvidia has their cuGraph library that boast some pretty impressive speedup on notoriously intensive graph problems too. It's not as mature as I'd like yet, but it is incredibly fast if you have some GPU compute available.

[Fixpoint]

changing a few lines, you can run it on multiple CPU cores, multiple servers, Google TPU, your video card or heck, you can run massive computation projects on AWS.

As far as I know, the Python language does not have any support for parallelization. The magic is done by the modules "threading" and "multiprocessing" which are just regular libraries. Just have a look at the source code and you'll see.

So, that is possible with virtually all other languages as well, C included. It has nothing to do with the Python language.

Actually, people do this, because the performance gain is immense.

Well, as I explained in great detail, this is in general not possible because the costs are way too high. If you can translate a project from one language into another, you are living in a very lucky special case. Your application is probably just arithmetics and well-understood.

you have to replace "import numpy" with "import cupy" ... and suddenly your code runs maybe 500 times faster

You can have a special case where this happens, yes, but in general, no. It is mathematically provable that this is in general a false statement.

And again: It is not a matter of Python but of a library. You can do that in each and every other language as well. If you are interested, I'll explain how.

[tszaboo]

changing a few lines, you can run it on multiple CPU cores, multiple servers, Google TPU, your video card or heck, you can run massive computation projects on AWS.

As far as I know, the Python language does not have any support for parallelization. The magic is done by the modules "threading" and "multiprocessing" which are just regular libraries. Just have a look at the source code and you'll see.

So, that is possible with virtually all other languages as well, C included. It has nothing to do with the Python language.

It is possible, however the amount of work to make some code working with parallel processing is 5 minutes on python and probably months on C.

Actually, people do this, because the performance gain is immense.

Well, as I explained in great detail, this is in general not possible because the costs are way too high. If you can translate a project from one language into another, you are living in a very lucky special case. Your application is probably just arithmetics and well-understood.

I gave you an example, why projects are re-written. You have to treat it as an example.

And again: It is not a matter of Python but of a library. You can do that in each and every other language as well. If you are interested, I'll explain how.

No, not really interested. Any time I start programming something, two things come to my mind:
1) This would be a lot simpler in python.
2) Why am I programming, I am an EE. I should be designing a PCB.

[Fixpoint]

Any time I start programming something, two things come to my mind:
1) This would be a lot simpler in python.

I understand that, for two reasons:

• Usually, a tool you know well is a tool that is efficient for you. Greybeards who have used C or FORTRAN all their life usually tell me "C/FORTRAN is a great language". Problem is, they live in a bubble.
• You say you are an EE, so that means that you are probably doing things of a certain nature, maybe computational or control stuff. Python has very strong library support for that. But from a wider software development perspective, there are more things to consider. Unfortunately, Python has weaknesses in those areas.

My problem with Python is the culture it is associated with.

In my life, I have met too many people who had a tendency to view the hardware part as "the project" and the software part as "the nonsense that has to be done for technical reasons but that nobody cares about". Python caters to this viewpoint of "just get it done and don't care", and that's why it frustrates me. The result is bad software and bad products. I think we all agree that a product is not good or safe just because it works somehow.

It's not Python's fault as such, but it just has come this way. I have learned to see Python as the leader of a "don't know, don't care" culture. So, maybe you understand where I come from and what my reservations are.

[Howardlong]

In my life, I have met too many people who had a tendency to view the hardware part as "the project" and the software part as "the nonsense that has to be done for technical reasons but that nobody cares about". Python caters to this viewpoint of "just get it done and don't care", and that's why it frustrates me. The result is bad software and bad products. I think we all agree that a product is not good or safe just because it works somehow.

Is that not because of Python, rather than despite Python? I mean, you don't go to Python to write bare metal, which is where us deeply embedded hardware people live. I like to think most of us care about performance and efficient use of resources. Most of that mindset goes out of the window with Python.

[Fixpoint]

Is that not because of Python, rather than despite Python? I mean, you don't go to Python to write bare metal, which is where us deeply embedded hardware people live. I like to think most of us care about performance and efficient use of resources. Most of that mindset goes out of the window with Python.

Yes, bare metal isn't done with Python, but there are other things that are done with Python that can still be safety-critical. I once worked in such a project, and in a meeting I pointed out some fundamental problems with the Python system (safety-critical!), but they didn't know AND didn't care. That's an example of the "don't know, don't care" culture I was talking about.

And the other thing: today's microcontrollers are so powerful and the software projects oftentimes so complex that it wouldn't be a good idea to optimize down to the individual clock cycle. If you have a 32-bit chip running at 20 MHz and 16, 32, or 64 KB of RAM, there is rarely an excuse anymore for bad software design (or the lack thereof). The chip is by far powerful enough to accommodate a few clock cycles and some bytes of overhead.

[Howardlong]

And the other thing: today's microcontrollers are so powerful and the software projects oftentimes so complex that it wouldn't be a good idea to optimize down to the individual clock cycle. If you have a 32-bit chip running at 20 MHz and 16, 32, or 64 KB of RAM, there is rarely an excuse anymore for bad software design (or the lack thereof). The chip is by far powerful enough to accommodate a few clock cycles and some bytes of overhead.

“a few clock cycles”.... if only it were. That’s the problem, performance is so often an afterthought, not part of the design criteria. I beleive that this stems from the mentality of separation of functional and non functional requirements, inevitably non functional requirements become afterthoughts.

A practical demonstration is to compare a Keysight Infiniivision scope to a Tek MDO scope. One is built from the ground up to perform well in all use cases, it’s in its DNA. The other one isn’t.

[Mattjd]

Whats faster a program that takes 8 months to write and finishes in 5 minutes or a program that takes 1 day to write and finishes in 5 hrs?

People like Python because its easy to learn, intuitive to write, and there is loads of open source community support for it. Why is there loads of open source community support though? Well that's because if you learn the lower level details of the language such as the CRO and Dunders then you can write very dynamic, portable, and minimalist code.

Duck typing is one of the best features of Python imo. Tell me C++ coders don't want it. Literally almost every C++ developer where I work just loves loves loves Templates. Wouldn't it be nice to not have to write a template for every possible type?

This is all a trade off people.

Python is realizing the need for static type checking, hence core Python is in full support type annotations of variables, functions, and classes. There are then non-definition time tools just as MyPy which will tell you if you are reassigning or passing an annotated variable incorrectly. If you're still stuck in 2.7 land, well sucks to be you and your company. Like I stated earlier, its not even supported anymore. I don't even think python 3.5 is being supported since earlier this year.


Again, if you don't like Python, don't use it.

Reading these posts, it really seems like what everyone wants is for Python to be written like Java. Go write Java then

[Howardlong]

Whats faster a program that takes 8 months to write and finishes in 5 minutes or a program that takes 1 day to write and finishes in 5 hrs?

People like Python because its easy to learn, intuitive to write, and there is loads of open source community support for it. Why is there loads of open source community support though? Well that's because if you learn the lower level details of the language such as the CRO and Dunders then you can write very dynamic, portable, and minimalist code.

Duck typing is one of the best features of Python imo. Tell me C++ coders don't want it. Literally almost every C++ developer where I work just loves loves loves Templates. Wouldn't it be nice to not have to write a template for every possible type?

This is all a trade off people.

Python is realizing the need for static type checking, hence core Python is in full support type annotations of variables, functions, and classes. There are then non-definition time tools just as MyPy which will tell you if you are reassigning or passing an annotated variable incorrectly. If you're still stuck in 2.7 land, well sucks to be you and your company. Like I stated earlier, its not even supported anymore. I don't even think python 3.5 is being supported since earlier this year.


Again, if you don't like Python, don't use it.

Reading these posts, it really seems like what everyone wants is for Python to be written like Java. Go write Java then

Absolutely right, it's horses for courses, with a sprinkling of bias to using the skillsets and tools that you have at hand.

[tszaboo]

In my life, I have met too many people who had a tendency to view the hardware part as "the project" and the software part as "the nonsense that has to be done for technical reasons but that nobody cares about". Python caters to this viewpoint of "just get it done and don't care", and that's why it frustrates me. The result is bad software and bad products. I think we all agree that a product is not good or safe just because it works somehow.

Is that not because of Python, rather than despite Python? I mean, you don't go to Python to write bare metal, which is where us deeply embedded hardware people live. I like to think most of us care about performance and efficient use of resources. Most of that mindset goes out of the window with Python.

That is not strictly the case. If you have some general purpose code, with lot of branches, creating and deleting variables, then yes, python will be slower. Sometimes this doesnt even matter, because the code is short, and a few microseconds makes no difference.

When it comes to disc read, computational tasks, database handling, etc... it can reach speeds that are very close to C/C++. It's all in the libraries, that were written and optimised by the experts. When you write code to do matrix multiplication, the underlying function is not going to be pure python code, it is going to call from a selection of optimised functions (maybe written in C / C++) one that will do the multiplication, and return the result. There is an overhead, but it is small.

And here is to consider something: Most C / C++ coders dont know the optimiser. They dont understand how branch prediciton works on modern CPUs. They dont know about cache sizes. Or they dont know how to optimise the code to several different architectures. There will be that champion programmer, that can do it. But an average python programmer, using the usual well understood libraries could write a faster code, with less bugs than an average C programmer.

Yes, bare metal isn't done with Python, but there are other things that are done with Python that can still be safety-critical. I once worked in such a project, and in a meeting I pointed out some fundamental problems with the Python system (safety-critical!), but they didn't know AND didn't care. That's an example of the "don't know, don't care" culture I was talking about.

Yes, safety critical... I mean, you are supposed to use the right screwdriver for the job, right? Project requirements not communicated well, and not understood well is not really a problem of the language. I'm not even sure, if you can write safety critical code in python in the first place, because you would need to prove that the interpreter is safe in the first place.  That is not done AFAIK.

[Howardlong]

In my life, I have met too many people who had a tendency to view the hardware part as "the project" and the software part as "the nonsense that has to be done for technical reasons but that nobody cares about". Python caters to this viewpoint of "just get it done and don't care", and that's why it frustrates me. The result is bad software and bad products. I think we all agree that a product is not good or safe just because it works somehow.

Is that not because of Python, rather than despite Python? I mean, you don't go to Python to write bare metal, which is where us deeply embedded hardware people live. I like to think most of us care about performance and efficient use of resources. Most of that mindset goes out of the window with Python.

That is not strictly the case. If you have some general purpose code, with lot of branches, creating and deleting variables, then yes, python will be slower. Sometimes this doesnt even matter, because the code is short, and a few microseconds makes no difference.

When it comes to disc read, computational tasks, database handling, etc... it can reach speeds that are very close to C/C++. It's all in the libraries, that were written and optimised by the experts. When you write code to do matrix multiplication, the underlying function is not going to be pure python code, it is going to call from a selection of optimised functions (maybe written in C / C++) one that will do the multiplication, and return the result. There is an overhead, but it is small.

And here is to consider something: Most C / C++ coders dont know the optimiser. They dont understand how branch prediciton works on modern CPUs. They dont know about cache sizes. Or they dont know how to optimise the code to several different architectures. There will be that champion programmer, that can do it. But an average python programmer, using the usual well understood libraries could write a faster code, with less bugs than an average C programmer.

Yes, bare metal isn't done with Python, but there are other things that are done with Python that can still be safety-critical. I once worked in such a project, and in a meeting I pointed out some fundamental problems with the Python system (safety-critical!), but they didn't know AND didn't care. That's an example of the "don't know, don't care" culture I was talking about.

Yes, safety critical... I mean, you are supposed to use the right screwdriver for the job, right? Project requirements not communicated well, and not understood well is not really a problem of the language. I'm not even sure, if you can write safety critical code in python in the first place, because you would need to prove that the interpreter is safe in the first place.  That is not done AFAIK.

In this case you're right, of course. About three or four years ago, I was talking to an old work colleague and was surprised to find they were using Python on their backend high frequency trading systems: of course, this wasn't for the real grunt low latency response stuff, it was for scripting and managing processes.

Now you mention databases: this is another bone of contention. There's been a huge shift over the past decade or so towards using ORM abstractions on databases such as Hibernate and Linq. While this might make the programmer's life easy, and is all fluffy and nice for the architects, when those abstractions don't perform in given use cases, it's difficult to create maintainable resolutions when the code is hidden away in the abstraction. It's a paradigm almost as old as the hills to rely on programmer apathy and optimiser telepathy! For a DBA, when a vendor says their new optimiser is so much better, it's a signal that, while there may be fewer performance edge cases, those that remain will be even harder to fix.

[Marco]

Your complaint would be equally valid for relational databases. If you're going to suffer opaque execution, might as well get pretty code.

It's only half the shift any way, there's also NoSQL in the other direction. Choice is great, except the choice of using SQL for web development in the last 3 decades ... that was a trillion dollar mistake.