The spread of "Functional Programming"

[paulca]

Java got "Functional interfaces", "Streams API" and "Lambda" support back in Java 8.

Like many things it was picked up, abused, miss-used because it was "novel".  Something new to learn and play with.  I personally hoped it would fade out as a fad in Java.  It didn't.

Now I am fine with FOP when it has a purpose that it's suited for.  Similarly with OOP.  What I am not fine with is people who would otherwise have written sensible, readable, working, testable, performant code, instead start producing cryptic nested, anonymous lambda delegate patterns everywhere... they start making all their collections immutable... soon it looks like a different language, full of "implicits" and code so uncoupled through implicits and scope absurdities that not even an IDE can follow usages.  You literally have to run the thing and debug it.

It stinks of Javascript.  The worst proponents of it are UI devs who spend a lot of time in Angular et. al.  JS.

Particular pet hates.

1.  Complete lack of understanding of the paradigm in the first place.  When I ask these people if their lamda is pure or not, they look at me blankly.  When I ask them what the difference between a "map" operation and a "forEach" operation is, they say, "There isn't one.".  Explain what the benefits of idempotency are...  never a good answer.

(Ideopotency for electronics folks.  A toggle button is NOT idempotent.  Having a separate ON button and an OFF button, they are idempotent.  It doesn't matter how many times you press the ON or OFF button the thing will always end up in the same state afterwards.  (Or thats the strawman)

2.  Complete disregard for the underlying requirements to allow such a coding style.  "They are highly optimised".  They will say.  "You do know that by making everything immutable you also make it completely and utterly useless for anything functional as ALL software does it mutate state!  In order to do anything functional from an immutable state you have to copy the state while you modify it or create entirely new state.  Including allocating all that lovely memory.

3. Loss of any clear understanding of the control flows involved. (map/foreach etc.).  So often multiple tasks that should all be in the "same loop" control flow end up in a dozen different serialised loops or Cartesian nested loops.  Performance loss... 90%.  Why loop over a collection in one place, when you can do it in a dozen different places in duplicate, right?

4. It reads like garbage. 

5. More widely the actual FOP implementations range from nothing buy syntactic sugar (python) through, a good attempt but awkward (java), to WTAF (scala/haskal).

It has it's places.  In those places it excels.  Event driven, data driven,  highly distributed, information theory-esk, analytics, data science etc.   There are some very good reasons we don't write general application code in Scheme, JS, Closure.  There is a reason these languages and paradigms died.  They are not fit for "general purpose programming".  They are niche languages which solve a subset of problems really well and absolutely suck at some basic others.  Even then that still has a purpose in it's place.  We still do those tasks.  Just stop trying to write everything that way.  IT DOESN'T WORK and reads like crap.

/rant.

[Siwastaja]

Slow down. Before going for the rant please explain what is this TLA.

I tried to Google it and landed on a Wikipedia page which also fails to describe what it is and goes straight into "history", making me believe this is some BS thing.

EDIT: opening post apparently edited to clarify what is being meant, it wasn't what I thought it was.

[paulca]

Functional Programming

https://en.wikipedia.org/wiki/Functional_programming

[paulca]

Take a look at the code side-by-side here:
https://en.wikipedia.org/wiki/Functional_programming#Imperative_vs._functional_programming

Code: [Select]

const numList = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let result = 0;
for (let i = 0; i < numList.length; i++) {
  if (numList[i] % 2 === 0) {
    result += numList[i] * 10;
  }
}

Okay....

Code: [Select]

const result = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
               .filter(n => n % 2 === 0)
               .map(a => a * 10)
               .reduce((a, b) => a + b, 0);

Reads like garbage.  Maybe it reads well to an information theorist or a mathmetician/data analyst, but to me, a meer software engineer, it reads like garbage and requires I actually put my brain i gear to work out what in hell it's doing.

EDIT:  Note the use of the tripple equality operator. It's the only place I have seen it is Javascript.  Usually meaning "Is both equivalent AND of the same type.  "1" == 1 is basically true in a very loosely typed language so they introduce "1" === 1 which is false (usually, it's JS, it depends). https://www.destroyallsoftware.com/talks/wat

[Tation]

In fact I find the 2nd syntax more readable and understandable…

Just a «classic» programmer here, from asm to C, to C++, to Python, definitely not a lover of «cool», «modern», «a la mode», bells and whistles.

[Siwastaja]

Reads like garbage.

Yeah, you need to get used to it. I like the Wikipedia comment, "might lead to development of more robust code that avoids certain issues that might arise when building upon large amount of complex, imperative code, such as off-by-one errors".

Sure you can get off-by-one, but usually problems are in corner cases and an imperative loop explicitly shows where the corners are. These "higher order" functions hide more and more from the programmer, really the only way to see what is happening is to test, writing test vectors as imperative loops and testing them.

At least you can mentally test the imperative loop while writing it to get an immediate understanding about corner cases.

[brucehoult]

Code: [Select]

const numList = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
let result = 0;
for (let i = 0; i < numList.length; i++) {
  if (numList[i] % 2 === 0) {
    result += numList[i] * 10;
  }
}

Okay....

Code: [Select]

const result = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
               .filter(n => n % 2 === 0)
               .map(a => a * 10)
               .reduce((a, b) => a + b, 0);

Code: [Select]

   x =: 1 + i.10
   +/ 10 * (0 = 2|x) # x
300

??

[golden_labels]

If you have a new hammer, everything looks like a nail. An opposite of the original phrase.

[DimitriP]

t, "might lead to development of more robust code that avoids certain issues that might arise when building upon large amount of complex, imperative code, such as off-by-one errors".

I took an ADA class once...and dropped it 3 class meetings later.
Looking at code should not necessarily fill you with happiness, but it shouldn't  make you wanna smash the keyboard on the monitor either !   [ or worse , go lookin' for the person that wrote it - and sometimes it's you! ]

[SiliconWizard]

FP - at least "real" FP - has a number of benefits (immutable everything, no hidden state, easier to prove correctness, ...), but some languages (like JS) have included it as an additional paradigm and that makes a whole mess.

Yes, one key issue with FP is that it largely hides execution flow, which can make developers very uncomfortable (and cause real issues in some applications).

[Marco]

The map reduce stuff at least is useful for parallel programming. Lambdas seem useful for those who really like one liners.

Monads are useful to make your code fundamentally inaccessible to 99% of other programmers. Thankfully most of the people who would have been into Haskell in the past seem to now choose Rust, I can learn a new syntax but monads break my tiny little mind.

[aeberbach]

Proponents tell me it isn't as inefficient as it looks, the call stack of > 50 functions I sometimes see in the now-useless debugger is "optimised out" or otherwise handwaved away. Not sure I believe them.

[Siwastaja]

FP - at least "real" FP - has a number of benefits (immutable everything, no hidden state, easier to prove correctness, ...)

That's indeed an appealing list until you realize that conventional imperative languages provide distributed state and mutable everything because that is what programmers need. And that correctness is difficult to prove because real-world problems are complex and poorly defined to begin with.

So this list reminds me of advantages you get by not owning a car. Which isn't helpful if you need a car.

For some specific types of projects functional programming (little bit of centralized state, simple data flows, heavily paralleled calculation) is probably a perfect match but this is difficult to predict in advance and working around the limitations is expensive. You either create a horrible mess or you refactor/redesign the codebase all the time for each new small feature needed.

Real-world experience overseeing a project also shows that functional programming leads to atrocious stuff like using database connections to store shared state because sharing it in-memory requires too much work, completely eliminating any possibilities for parallelization. Proponents say this drives continuous refactoring preventing "temporary" small hacks which become permanent but it really does not have this effect. Instead it drives "temporary" large hacks that become permanent.

[paulca]

The map reduce stuff at least is useful for parallel programming. Lambdas seem useful for those who really like one liners.

It's where I have 3 years experience in FP.  Using Apache Spark on petabytes of data with a 3000 node cluster.

The immutability permits high parallelism.  Spark's flow works backwards, (declarative) it starts at the result in you want and the first thing is does is "raise" a "RRD Missing" for it.  So it goes through your "operation plan" and looks for the function which generates that RDD.  Of course it requires an input RDD and so it cascades back to the start of the flow.

The interesting thing is... if your last statement is something like

theDataSet.take(10);

Spark, assuming you don't sort the data, will only LOAD the first 10 records from your input dataset.  It's smart and it's extremely lazy.

However.  There were a few teams using Spark as FP programmers would.  They never made any progress because not enough people understood what it was doing and it never worked because it always "coupled" data and broke distribution of it.  "collectAsList()" and things pulling it all local to one node etc. etc.

What we did was to abuse the "map" function to insert "Plain old Java" code segments into Spark flow.  So the FP and Spark smart devs did all the wiring and flow control, but sensible sized chunks of functionality was encapsulated in plain old Java wrapped in a "MapFunction" interface.

Therefore we could use both paradigms and maximise the output and quality using the resources (people) we had.

The flak we got was "by putting 10 java operations into a Map function, you are losing the potential of those 10 being parallelised."

However, they never mastered their parallelism, data locallity, nor maintaining it across the full flow... so while their processing took 4 hours to complete, like ours did, once we mastered how to keep the data distributed our time dropped to 20 minutes.

[paulca]

One of the patterns that "irks" me most is their insistence on "Anonymous functions".

When coupled with first order functions which get passed and returned by other functions... and they are anonymously defined....   they go the final obfuscation and name all the variables/symbols a, b, x, y.  (Look.  I get it.  A lot of people came through the Maths faculaty to get into computing, but listen carefully.  IT IS NOT ALGEBRA!  Not even close.  Stop trying to make it so.

InteliJ can't even follow these calls.

It's similar to C++ code converted into C code using void*(**) stuff for everything.  Then chaining it together into completely anonymous function calls passing void pointers and taking void pointers as returns.

It's very reminisent, though not pleasant, of working in Angular.  Although, again, there FP has at least some purpose.  UIs are typically laid out as a tree of containers and sub containers and eventually UI elements within.  The fact browsers can render parts of the page in parallel ... and the whole UI "Event propogation" stuff kinda fits the flow a bit.

What I hate though is that everything is a function, which takes a function as it's argument and returns a function which takes a function as it's argument.  Everything is recursive.  It's just not a nice way to code.  At least name the damn functions!

[dferyance]

I've worked with ReactJS programmers who don't understand the functional programming ideas react is based on. There are things about ReactJS that are problematic, but I appreciate the functional approach.

I am convinced that functional programming in imperative languages is always going to be problematic. There is too much history of mutable state that even if some programmers understand it there will always be some that don't.

[Siwastaja]

The map reduce stuff at least is useful for parallel programming. Lambdas seem useful for those who really like one liners.

It's where I have 3 years experience in FP.  Using Apache Spark on petabytes of data with a 3000 node cluster.

The immutability permits high parallelism.

Yeah, sure. And we chose a functional language for a project partially for this "high scalability". And here we are, doing continuous work on performance improvements, this work started when we had our 200th customer, and we are talking about maybe a gigabyte per day total traffic stuff.

Parallel execution is worthless if bottlenecks are IO, state management through SQL database and stuff like that. A stupidly simple loop in C iterating everything sequentially on a single CPU core would be 1000 times faster.

Once you can architect everything perfectly, sure that enables parallelization, but you can get parallelization in any imperative language as well by understanding what you do. And from what I have seen in real world projects, functional languages do not drive you into the parallelizable design automagically at all, in fact opposite is true since it forces programmers with real-world time/budget limits to work around the limitations and e.g. replace shared memory access with IO which is much worse than shared memory access.

[paulca]

I've worked with ReactJS programmers who don't understand the functional programming ideas react is based on. There are things about ReactJS that are problematic, but I appreciate the functional approach.

I am convinced that functional programming in imperative languages is always going to be problematic. There is too much history of mutable state that even if some programmers understand it there will always be some that don't.

Explain two things to me.

How do you produce any measurable output effect of your software if you can't mutate anything?

What do you think the CPU does?  When you chain a bunch of function calls together, where do you think the state actually goes?  Yes.  It is mutated into and out of memory.

The emperor is NAKED!

[5U4GB]

If you have a new hammer, everything looks like a nail.

Almost.  It's "if you have a new hammer, everything looks like a thumb".

[5U4GB]

Parallel execution is worthless if bottlenecks are IO, state management through SQL database and stuff like that. A stupidly simple loop in C iterating everything sequentially on a single CPU core would be 1000 times faster.

Code: [Select]

SQLExecute( "SELECT * FROM table" );
for ( count = 0; SqlRead() != SQL_ERROR && count < INT_MAX; count++ );
return count;

[DiTBho]

I stopped writing ErLang code.

It is very performant in parallel things, but, making a "social engineering" consideration, it is also damn unproductive because there is never anyone willing to help you. I don't know why, but it seems that FP does not appeal to many people. I don't know, but most people stay away from it

So few people on many things to write, that's why I say "not very productive", although in some contexts it would be the best solution

[dferyance]

How do you produce any measurable output effect of your software if you can't mutate anything?

In the case of ReactJS, you don't do the mutations in your code. Instead you return from your function how you want the HTML document to look like. ReactJS itself mutates the state of the DOM to match. It is very nice from a theoretical perspective, but updating the DOM isn't exactly the hard part of web programming. So I'm not sure it is actually all that helpful in practice.

So yes, of course there is imperative code. However, the part of the system to determines what the page should look like can be purely functional. It takes experience to know the best way to design a system, instead of just following that everything must be OOP, everything must be imperative or everything must be functional.

I think the parallelism advantages of function programming are way overstated. Its completely possible to write imperative code that runs in parallel. Its also possible to use mutability but just not in a way to cause problems with parallelism. Fundamentally, designing parallel systems is a very difficult problem and it cannot be solved just by functional programming.

[paulca]

Literally seen FP blow up today.

The absolute "classical".... Stack Overflow.

DB Driver.  Using recursion instead of iteration to empty a command buffer.  Only the DB server was having a little lie down at the time.  The result was the stack overflow after it spun around in it's recursion of "need()->write()->begin()->need()->write()->begin()->need()->write()->begin()->need()->write()->begin()->need()->write()->begin()->need()->write()->begin()->need()->write()->begin()->need()->write()->begin()"

Recursion is neat.  Here is a tip though.  Do not use recursion on unbounded operations.... like infinite retries.  If you know the max breadth and depth of your operation, go ahead if recursion works for you, I mean it IS the preferred way for many things, but NOT all.  Using recursion when plain old iteration would have been just as good, NO, just NO. 

EDIT:  I was going to suggest looking at ASM to understand why iteration would be preferred over recursion.  However I had to stop and correct myself, because most of the pain of the stack in recursion actually comes from the C binary format and how "function calls" work with the stack.  If you are in pure ASM, there is no absolute need to use the stack to do recursion.  You can do "in frame" recursion, as long as you don't need to leave any breadcrumb trail for cascading output etc.  In ASM you can just two "blocks" of code, instead of functions and ping pong back and forth "jmp wait for data"...  "jmp write data"... until X..  jmp back "out".  No need to push the return address to the stack as it doesn't need to be recorded.  You could argue it's just "GOTO" based iteration, not recursion, but then we would need to get teh big dictionary out and fight over all the finer details of the terms.

[brucehoult]

Using recursion when plain old iteration would have been just as good, NO, just NO. 

If it really is an iteration then any decent modern compiler should optimise the tail-calls.

Relevant paper from 47 years ago in 1977:

https://en.wikisource.org/wiki/Lambda:_The_Ultimate_GOTO

Code: [Select]

#include <stdio.h>

typedef int status;

status need(void *params);
status write(void *params);
status begin(void *params);

status need(void *params) {
  printf("in need\n");
  return write(params);
}

status write(void *params) {
  printf("in write\n");
  return begin(params);
}

status begin(void *params) {
  printf("in begin\n");
  return need(params);
}

int main(){
  int param;
  need(&param);
  return 0;
}

Compiled with ...

Code: [Select]

debian@lpi4a:~$ gcc -Os fp.c -o fp
debian@lpi4a:~$ ./fp | head
in need
in write
in begin
in need
in write
in begin
in need
in write
in begin
in need
debian@lpi4a:~$ 

The machine code:

Code: [Select]

00000000000005d0 <main>:
 5d0:   1101                    add     sp,sp,-32
 5d2:   0068                    add     a0,sp,12
 5d4:   ec06                    sd      ra,24(sp)
 5d6:   102000ef                jal     6d8 <need>
 5da:   60e2                    ld      ra,24(sp)
 5dc:   4501                    li      a0,0
 5de:   6105                    add     sp,sp,32
 5e0:   8082                    ret
        ...

000000000000069c <begin>:
 69c:   1141                    add     sp,sp,-16
 69e:   e022                    sd      s0,0(sp)
 6a0:   842a                    mv      s0,a0
 6a2:   00000517                auipc   a0,0x0
 6a6:   05e50513                add     a0,a0,94 # 700 <_IO_stdin_used+0x8>
 6aa:   e406                    sd      ra,8(sp)
 6ac:   f15ff0ef                jal     5c0 <puts@plt>
 6b0:   8522                    mv      a0,s0
 6b2:   6402                    ld      s0,0(sp)
 6b4:   60a2                    ld      ra,8(sp)
 6b6:   0141                    add     sp,sp,16
 6b8:   a005                    j       6d8 <need>

00000000000006ba <write>:
 6ba:   1141                    add     sp,sp,-16
 6bc:   e022                    sd      s0,0(sp)
 6be:   842a                    mv      s0,a0
 6c0:   00000517                auipc   a0,0x0
 6c4:   05050513                add     a0,a0,80 # 710 <_IO_stdin_used+0x18>
 6c8:   e406                    sd      ra,8(sp)
 6ca:   ef7ff0ef                jal     5c0 <puts@plt>
 6ce:   8522                    mv      a0,s0
 6d0:   6402                    ld      s0,0(sp)
 6d2:   60a2                    ld      ra,8(sp)
 6d4:   0141                    add     sp,sp,16
 6d6:   b7d9                    j       69c <begin>

00000000000006d8 <need>:
 6d8:   1141                    add     sp,sp,-16
 6da:   e022                    sd      s0,0(sp)
 6dc:   842a                    mv      s0,a0
 6de:   00000517                auipc   a0,0x0
 6e2:   04250513                add     a0,a0,66 # 720 <_IO_stdin_used+0x28>
 6e6:   e406                    sd      ra,8(sp)
 6e8:   ed9ff0ef                jal     5c0 <puts@plt>
 6ec:   8522                    mv      a0,s0
 6ee:   6402                    ld      s0,0(sp)
 6f0:   60a2                    ld      ra,8(sp)
 6f2:   0141                    add     sp,sp,16
 6f4:   b7d9                    j       6ba <write>

I can let that run all day without stack overflow.

[brucehoult]

Ok, you edited while I was writing my reply.

EDIT:  I was going to suggest looking at ASM to understand why iteration would be preferred over recursion.  However I had to stop and correct myself, because most of the pain of the stack in recursion actually comes from the C binary format and how "function calls" work with the stack.  If you are in pure ASM, there is no absolute need to use the stack to do recursion.  You can do "in frame" recursion, as long as you don't need to leave any breadcrumb trail for cascading output etc.

If there are any "breadcrumbs" then you COULD NOT write it as a loop even if you wanted to. Not without allocating the breadcrumbs on some user-created stack or linked list, which will overflow just the same as an infinite recursion will.

If by "the C binary format and how "function calls" work with the stack." you mean x86, that is no obstacle.

Exactly the same program compiled with exactly the same gcc -Os fp.c -o fp gives:

Code: [Select]

0000000000001080 <main>:
    1080:       f3 0f 1e fa             endbr64
    1084:       48 83 ec 18             sub    $0x18,%rsp
    1088:       64 48 8b 04 25 28 00    mov    %fs:0x28,%rax
    108f:       00 00 
    1091:       48 89 44 24 08          mov    %rax,0x8(%rsp)
    1096:       31 c0                   xor    %eax,%eax
    1098:       48 8d 7c 24 04          lea    0x4(%rsp),%rdi
    109d:       e8 3b 01 00 00          call   11dd <need>
    10a2:       48 8b 44 24 08          mov    0x8(%rsp),%rax
    10a7:       64 48 2b 04 25 28 00    sub    %fs:0x28,%rax
    10ae:       00 00 
    10b0:       74 05                   je     10b7 <main+0x37>
    10b2:       e8 b9 ff ff ff          call   1070 <__stack_chk_fail@plt>
    10b7:       31 c0                   xor    %eax,%eax
    10b9:       48 83 c4 18             add    $0x18,%rsp
    10bd:       c3                      ret
    10be:       66 90                   xchg   %ax,%ax
...

00000000000011a9 <begin>:
    11a9:       f3 0f 1e fa             endbr64
    11ad:       53                      push   %rbx
    11ae:       48 89 fb                mov    %rdi,%rbx
    11b1:       48 8d 3d 4c 0e 00 00    lea    0xe4c(%rip),%rdi        # 2004 <_IO_stdin_used+0x4>
    11b8:       e8 a3 fe ff ff          call   1060 <puts@plt>
    11bd:       48 89 df                mov    %rbx,%rdi
    11c0:       5b                      pop    %rbx
    11c1:       eb 1a                   jmp    11dd <need>

00000000000011c3 <write>:
    11c3:       f3 0f 1e fa             endbr64
    11c7:       53                      push   %rbx
    11c8:       48 89 fb                mov    %rdi,%rbx
    11cb:       48 8d 3d 3b 0e 00 00    lea    0xe3b(%rip),%rdi        # 200d <_IO_stdin_used+0xd>
    11d2:       e8 89 fe ff ff          call   1060 <puts@plt>
    11d7:       48 89 df                mov    %rbx,%rdi
    11da:       5b                      pop    %rbx
    11db:       eb cc                   jmp    11a9 <begin>

00000000000011dd <need>:
    11dd:       f3 0f 1e fa             endbr64
    11e1:       53                      push   %rbx
    11e2:       48 89 fb                mov    %rdi,%rbx
    11e5:       48 8d 3d 2a 0e 00 00    lea    0xe2a(%rip),%rdi        # 2016 <_IO_stdin_used+0x16>
    11ec:       e8 6f fe ff ff          call   1060 <puts@plt>
    11f1:       48 89 df                mov    %rbx,%rdi
    11f4:       5b                      pop    %rbx
    11f5:       eb cc                   jmp    11c3 <write>

No difference.