Author Topic: Neat algorithms  (Read 54921 times)

0 Members and 2 Guests are viewing this topic.

Offline gnif

  • Administrator
  • *****
  • Posts: 1937
  • Country: au
  • Views and opinions are my own
Re: Neat algorithms
« Reply #100 on: August 21, 2019, 04:50:39 am »
Just a little useful function I wrote that I call upon often...

Code: [Select]
#include <stdlib.h>
#include <stdio.h>

char * bytesToHR(unsigned long bytes)
{
  static const char * suffix[] = {"B", "KiB", "MiB", "GiB", "TiB", "PiB", NULL};
  const char ** s = suffix;
  float b = (float)bytes;

  while(b > 1023.0f && *(s+1))
  {
    b /= 1024.0f;
    ++s;
  }

  int len = snprintf(NULL, 0, "%.2f %s", b, *s);
  char * out = malloc(len + 1);
  sprintf(out, "%.2f %s", b, *s);
  return out;
}

int main(int argc, char * argv[])
{
  char * s = bytesToHR(12345678);
  printf("%s\n", s);
  free(s);

  return 0;
}

Output:

Code: [Select]
11.77 MiB

License: Free for general use anywhere, no attribution required.
 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4642
  • Country: us
Re: Neat algorithms
« Reply #101 on: August 21, 2019, 05:49:53 am »
Quote
tell me why recursion isn't allowed for safety critical code in automotive and avionics applications, and I have never been authorized to use it.
Well, you take fears that were probably relevant 30+ years ago, put them together into a "standard" that dates back 20+ years, and then hand the "approval for exceptions" process off to "management" that hasn't been following the science, or the art, for about the same length of time.  "No one ever gets fired for failing to approve "risky" code"...  (not that they ever really realize that it's probably far less risky to take a modern and proven recursive, dynamic-allocation, recursive library and use it, than to ask their mid-level engineers to achieve the same thing from scratch without "modern techniques.")   Sigh.
And those Arianes and 787s and F35s and Zumwalts and SLS all come in under budget and work perfectly as a result.  Advanced medical equipment has gotten better, cheaper, and more available each year as Moore's Law advances the underlying technology!  Hurah!   Grr.


Quote
We fill the stack area with a known pattern 0xdeadbeaf, and we check how many times the pattern appears after the algorithm has completed its execution  ;D
This works for a given environment and it cannot be written in a formal test-report without a manual justification, but it gives us useful information during the engineering steps.
What?  Only during test?  We had that check in production code, for every process (recursive or not), at least "periodically" (I forget whether this was a "background" check, or something that happened in the scheduler.)  Of course, that WAS in an environment where probably the worst that would happen is irate customers would call up with "it rebooted and the last thing it said was something about "process has low stack."  Please fix it!"...)
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: Neat algorithms
« Reply #102 on: August 21, 2019, 10:07:30 am »
Well, you take fears that were probably relevant 30+ years ago, put them together into a "standard" that dates back 20+ years, and then hand the "approval for exceptions" process off to "management" that hasn't been following the science, or the art, for about the same length of time. 

So 200 guys employed on daily (testing, development, documentation, QA compliance, etc) activities in avionics are all wrong, as well as 20 senior-engineers with 30 years of experience  :-//

I am usually the rebel of the group, but we have a military approach to seniors, so I trust them, even if it's not clear why they give directives

  • MISRA-C, Rule 141, functions shall not call themselves, either directly or indirectly
  • SPARK-ADA, Rule 17.2, functions shall not call themselves, either directly or indirectly


What?  Only during test?

It's clearly written that we take this approach during the early steps of development, I can add,  "when" we cannot access ICEs or other equipment, or when we quickly need to have a confirmation. We have to run the full testing session only later, after the first interaction of the engineering phase has completed.

  • step0: project commission -> requirements, constraints, etc (here there are more managers, customers with their lawyers, all involved in private talks than engineers and technical staff)
  • step1: engineers receive the commission, and they start the first interaction of the engineering phase (draft-r0)
  • step2: guys in the testing-squad write test-cases and test things. The first interaction of the testing phase ends with a report about problems, bugs, things that need to be fixed/improved
  • step3: second interaction of the engineering phase (draft-r1)
  • step4: second iteraction of the testing phase -> problems, bugs, things that need to be fixed/improved
  • ...
  • ...
..

When the step(n) is PASSED
  • step(n+1): making the code MISRA & DO178B compliant
  • step(n+2): testing everything again, and performing things like the dynamic coverage
  • step(n+3): QA guys check for MISRA & DO178B compliance, and checks test-report's logs

When the step(m) is PASSED the project gets committed to Doors with a progressive revision {A, B, C, ...}, otherwise it's still DRAFT-{r0, r1,r2,...} and it goes back to engineers and testing-squad.

-

The "artisanal" approach I was talking about is during draft-r0, but it's also what I usually use at home, where I do not yet have a BDI2000/3000 able to monitor the SP register, or any special hardware able to monitor watchpoints (physical addresses to ram), or something able to track triplets (this is usually done for dynamic coverage and costs 20k euro in software and hardware, but it can upload triplets at run-time up to 60Mbyte/sec on a firewire link) so I sometimes use the same approach via a super-cheap gdb-stub on a super-cheap serial line @ 115200bps.
« Last Edit: August 21, 2019, 11:24:09 am by legacy »
 

Online NorthGuy

  • Super Contributor
  • ***
  • Posts: 3516
  • Country: ca
Re: Neat algorithms
« Reply #103 on: August 21, 2019, 12:35:25 pm »
So 200 guys employed on daily (testing, development, documentation, QA compliance, etc) activities in avionics are all wrong, as well as 20 senior-engineers with 30 years of experience  :-//

You wouldn't argue that nosediving two Boeings into the ground was the right think to do, would you?

And they cannot even find the bug ...

Perhaps having too much bureaucratic rules, and thereby sifting out all the employers who wanted to use their own mind, has something to do with that.
 

Offline RoGeorge

  • Super Contributor
  • ***
  • Posts: 8453
  • Country: ro
Re: Neat algorithms
« Reply #104 on: August 21, 2019, 02:10:52 pm »
Or, perhaps NOT obeying all those rules will nosedive into the ground 10 airplanes daily.
Anecdotal references are not a proof.

Unless somebody comes up with something better, let's not forget Aeronautics is the safest transportation industry.

Online NorthGuy

  • Super Contributor
  • ***
  • Posts: 3516
  • Country: ca
Re: Neat algorithms
« Reply #105 on: August 21, 2019, 02:45:40 pm »
Or, perhaps NOT obeying all those rules will nosedive into the ground 10 airplanes daily.
Anecdotal references are not a proof.

Exactly. All we know is that with these rules in place, the software can crash a plain without giving its human pilot any chance.
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17781
  • Country: fr
Re: Neat algorithms
« Reply #106 on: August 21, 2019, 03:00:15 pm »
Just consider the funny example I gave about strict rules for risk mitigation. All we can say they do at best is avoid some specific cases of fuck-up. Not that the design will ever be problem-free. Obviously doesn't mean we shouldn't follow rules. Basic logic at play here.

Admittedly, they can tend to give a false sense of safety though, and that's when they could be considered counter-productive. It's very hard to determine whether the statistical net outcome for any given rule (or a set of them) is positive or negative. Good luck with that. So the usual approach in safety-critical settings is to enforce as many as possible, close to, but just below the threshold at which they would completely halt development, and then hope for the best.

@RoGeorge said:
Quote
Anecdotal references are not a proof.
which may be either completely right, or false. Depends on what you consider here. Even just one occurence of a problem is complete proof that the problem CAN happen, and in safety-critical settings, this should be enough to take action to improve things based on that single occurence only. Constant improvement.

Could mean that the processes at play here have holes. Whether this is due to too many rules, or not enough, or whatever else is extremely difficult to determine. Needs very thorough analysis. But it definitely NEEDS to be analyzed IMO, and actions taken if at all possible.


 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: Neat algorithms
« Reply #107 on: August 21, 2019, 03:50:55 pm »
Admittedly, they can tend to give a false sense of safety though

It's not correct and it sounds rather misleading. Concerning software (the hardware follows different directives, although similar), the whole DO178B/level{A..D} describes how the engineering and testing activities have to be done, while the MISRA-C and SPARK-ADA tell us some rules to manage the job.

This doesn't magically make the "source" absolutely safe and "bug-free", but it rather helps people in a large team to cooperate in a productive and deterministic way with all the tools and resources known and described on a piece of paper so we can learn something new each time to improve the product during its life-cycle.

Our directives do not only consider the source, but rather even how it's tested concerning equipment, procedures and human resources, so if a senior says that recursion shall be avoided, he/she knows that it might work, but it will be more difficult to manage within the software life cycle.
« Last Edit: August 21, 2019, 04:07:00 pm by legacy »
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: Neat algorithms
« Reply #108 on: August 21, 2019, 04:25:13 pm »
I've never used recursion in any critical piece of "embedded" code even in settings where this was not a rule. A mix of common sense, risk mitigation and often no real need for/benefit of recursion for rather low-level stuff. I've never set a "no recursion" rule though when it was not strictly required from a regulatory POV.

Rules say no recursion. This is a guide, but it's not written in stones. If you want to use recursion you have to
  • isolate the module in the Door's traceability, so it's clearly and formally marked as under your responsibility
  • provide a document to demonstrate the benefit of the approach (why is it better?), and pros vs cons
  • provide a detailed description of the special test-report to demonstrate things, concerning the equipment to be used, the testing procedure for the testing squad and how to perform analysis
  • submit a formal request of approvation and obtain at least three confirms by seniors
  • when approved, they will ask for a preliminary analysis of the error propagation and behavior on abnormal conditions (how will behave on failure?)
  • if approved, you have submit a formal request of approvation even to the QA-squad, which will archive documents if compliant to the internal procedure

Something similar has been even approved for the AFDX switch (it's a kind of ethernet switch, made deterministic),  but it was hard to obtain the OK.
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 17781
  • Country: fr
Re: Neat algorithms
« Reply #109 on: August 21, 2019, 05:33:19 pm »
Admittedly, they can tend to give a false sense of safety though

It's not correct and it sounds rather misleading. Concerning software (the hardware follows different directives, although similar), the whole DO178B/level{A..D} describes how the engineering and testing activities have to be done, while the MISRA-C and SPARK-ADA tell us some rules to manage the job.

What exactly is not correct and sounds misleading? I think you're consistently missing the point of the discussion.
I was considering the fact that blindly following some hard rules CAN tend to give a false sense of safety. CAN. I didn't say it always did.
Then a "false sense of safety" means that engineers would think those rules would protect them from ever fucking up. WHEN that's the case (not saying it is ALWAYS the case, but it does happen), it CAN actually be counter-productive (not saying it is ALWAYS the case).

Read everything again. That didn't mean no rules should be followed either. Argh, logics...

Our directives do not only consider the source, but rather even how it's tested concerning equipment, procedures and human resources, so if a senior says that recursion shall be avoided, he/she knows that it might work, but it will be more difficult to manage within the software life cycle.

Again, this point is very real, but moot in the context of this discussion. As I said as a preliminary, if you don't have a choice, discussing something is interesting, but completely pointless.
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22435
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Neat algorithms
« Reply #110 on: August 21, 2019, 06:34:16 pm »
The false sense comes when, because you're surrounded by rules for every step of development, you may get into the assumption that they are actually comprehensive and therefore you have no responsibility yourself.

Well, as a coder you may not necessarily have any responsibility yourself, but it applies recursively (hah) to your immediate manager, or QA, and so on up the chain of responsibility.

The better case seems to be, having very experienced coders -- who understand the edge cases of the algorithm in question, and the pitfalls of the language -- and also subjecting the resulting product (at a source code level, as well as at a physical pen-test level, say) to a battery of tests by researchers even more experienced in picking apart edge cases.

But it's also not clear to me if such an approach is likely to have any better economy (that is, a higher 1/(bugs * dollars) figure-of-merit) versus the traditional aerospace approach.  Both are onerously expensive (and for good reason!).  I'm no manager, I'm just speculating.

Heh, or even necessarily that that's an adequate metric, since it may well be that mainstream, cheap and buggy development is so much cheaper that it actually offers a better score.  In which case we might choose a different exponent for the parameters, or we would want to stipulate that the bug rate has to be below some maximum.  (Incidentally, the bug rate obviously can't be measured at approval or release time, but it can be measured over the lifetime of the product, say in the number of bugs noted by operators and maintenance, and the number of fixes pushed to the field.)

(Heh, well, maybe that's not even a good idea at all since it attaches a negative monetary value to bug disclosure and patching.)


Well, you take fears that were probably relevant 30+ years ago, put them together into a "standard" that dates back 20+ years, and then hand the "approval for exceptions" process off to "management" that hasn't been following the science, or the art, for about the same length of time. 

So 200 guys employed on daily (testing, development, documentation, QA compliance, etc) activities in avionics are all wrong, as well as 20 senior-engineers with 30 years of experience  :-//

Just to tickle the dragon here -- yes, that may be a reasonable argument.  It would be foolish after all to expect that computer science is over, done and done, fully researched.  CS is rather high level but it often trickles down into engineering.

Not to say that the existing process is bad in-and-of-itself, or compared to a particular hypothetical.  Just that, compared to some unknown hypothetical best, it's probably not there, and there is always room for improvement.

Which is the more general argument, from safety culture: there's always room for improvement. :)

It's not clear how much is actually in flux (and whether it's changing truly for the better, or because office politics), or if it's literally been unchanged for 30 years because bureaucracy, so, this covers the bases. :-//

Tim
« Last Edit: August 21, 2019, 06:38:01 pm by T3sl4co1l »
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Online NorthGuy

  • Super Contributor
  • ***
  • Posts: 3516
  • Country: ca
Re: Neat algorithms
« Reply #111 on: August 21, 2019, 06:56:18 pm »
... so it's clearly and formally marked as under your responsibility ...

Everything you do or accept is your responsibility. Bureaucratic rules often pretend to absolve people of responsibility. This makes people feel safe, irresponsible perhaps. But this doesn't really work. For example, that guy who made a mistake which caused two Boeing crashes. He is not going to be hanged in front of the crowd. But the rest of his life will be a nightmare where every minute he tries hard to convince himself that he didn't kill 500 people, that he is a good man who just followed rules.

Wouldn't it be better if he recognized from the start that the responsibility is his?
 

Offline RoGeorge

  • Super Contributor
  • ***
  • Posts: 8453
  • Country: ro
Re: Neat algorithms
« Reply #112 on: August 21, 2019, 07:06:04 pm »
Can we please return to the "Neat algorithms", topic?
 
The following users thanked this post: BravoV

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: Neat algorithms
« Reply #113 on: August 21, 2019, 09:13:20 pm »
Everything you do or accept is your responsibility.

It's a different kind of responsibility because in that case you temporarily become a sort of team leader regarding the project, and this means that you have to somehow coordinate people around you, as well as other people by emails and video conference. In short, since they see your name on Doors, you are the person who will be called for everything concerning the project.
 

Offline legacy

  • Super Contributor
  • ***
  • !
  • Posts: 4415
  • Country: ch
Re: Neat algorithms
« Reply #114 on: August 21, 2019, 10:21:47 pm »
What exactly is not correct and sounds misleading? [...] Then a "false sense of safety" means that engineers would think those rules would protect them from ever fucking up.

Precisely this point is incorrect and lets it pass that we should be a sort of monkeys to blindly follow hard rules.

You miss that things are really so complex that no-one can judge other's competence, and there is not either any "false sense of safety", here we need rules to handle a large group composed of people with different skills and capabilities, developers, testers, QA guys, guys who only study the error propagation on abnormal condition, etc, and each of them is no less important than the developer-squad. So each feedback is collected and things are done with care by planners, and their rules come from the experience about how to reduce friction and to facilitate the collaboration between each squad.

That's their competence, and this is also the reason why if you want to use the recursion, well ... it's not denied, but you have to open a branch on Doors, and if your approach will be successful, it will be applied again, and the rule revisioned/removed.

Note, there were attempts to use recursion, and ... attempts were not so successful in the last 10 years, in fact, SPARK2014 confirmed the rule, and recursion still shall be avoided. Which is the reason why you still need to open a branch on Doors.
 

Online NorthGuy

  • Super Contributor
  • ***
  • Posts: 3516
  • Country: ca
Re: Neat algorithms
« Reply #115 on: August 21, 2019, 10:31:31 pm »
Everything you do or accept is your responsibility.

It's a different kind of responsibility because in that case you temporarily become a sort of team leader regarding the project, and this means that you have to somehow coordinate people around you, as well as other people by emails and video conference. In short, since they see your name on Doors, you are the person who will be called for everything concerning the project.

I wouldn't call this a different kind.

If you organize and coordinate people, you're responsible for the results. If the effort fails, you'll be the one to blame, sack or whatever. You cannot have freedom without responsibility.
 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4642
  • Country: us
Re: Neat algorithms
« Reply #116 on: August 21, 2019, 11:11:53 pm »
Quote
Can we please return to the "Neat algorithms", topic?

The IP checksum has some interesting properties.
In theory, it's a one's complement checksum of 16bit words.  On a twos-complement ALU (as found in almost every CPU, ever), you can implement a 1's complement by using "end-around carry" - after you do a normal add, you add in any resulting carry bit:
Code: [Select]
  carry, Rd = Rs + Rs             ;; all Rx and math operations are 16bits, 2's comp.
  if (carry) Rd = Rd + 1;
But this would be relatively slow, and addition is nicely commutative, and addition is commutative and associative, and 1's comp addition is associative WRT byteswapping, so IP checksum functions usually get highly optimized.
On a generic 32bit CPU, the usual optimization is to accumulate the carries in a larger word and the possibly "fold" and add them at the end:
Code: [Select]
    for (word16 in words) {
      ck32 = ck32 + word16
    }
    while (ck32 >> 16) {
      ck32 = (ck32 & 0xFFFF) + (ck32 >> 16)
    }
But that's usually pretty silly on an 8bit CPU, and if you have both a CPU and language that allow access to the carry bit (for multi-precision math), you can do a lot better doing the end-around carry together with adding the NEXT byte:
Code: [Select]
    prevcarry = 0
    for () {
      prevcarry, ck8low = ck8low+nextbyte+prevcarry   // (and increment to following byte)
      prevcarry, ck8hi = ck8hi+nextbyte+carry
    }
    prevcarry, ck8low = ck8low + prevcarry          // final end-around carry
    ck8hi = ck8hi + prevcarry
Note that you'll need some sort of looping code that doesn't destroy the prevcarry bit...

So for example on AVR, while the usual lwip code will use the generic 32bit algorithm, you can write code that's less than half the size and more than twice the speed: https://github.com/WestfW/Duino-hacks/blob/master/ipchecksum_test/ipchecksum_test.ino

 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4642
  • Country: us
Re: Neat algorithms
« Reply #117 on: August 21, 2019, 11:21:53 pm »
Quote
If you want to use recursion you have to [six steps, including "approval by 3 seniors" and "the QA team"
Exactly.  It's so much easier to just say "no."
I'll bet that the numerous Computer Science texts and papers that have touted and analyzed the crap out of the recursive algorithms don't count much toward the justification, either...
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 8349
  • Country: fi
    • My home page and email address
Re: Neat algorithms
« Reply #118 on: August 22, 2019, 03:33:13 am »
Anyone interested in efficient RGB blending?

While this is not an algorithm per se, more like bit ops tips, the underlying idea — SIMD-like operations on unsigned integers with field width sufficient for intermediate results, and unpacking packed numeric fields for this by shifting every second field up by a full record width — is an interesting method; close enough to a neat algorithm to be noted here, IMHO.



If you have two ARGB colors as 32-bit unsigned integers, you only need five bit shifts, six binary and operations (that use one of two 32-bit masks), one binary or operation, and two additions, to calculate their 50% blend:
Code: [Select]
static inline uint32_t argb8888_blend_half(const uint32_t argb1, const uint32_t argb2)
{
    const uint32_t  rb1 = (argb1 << 7) & 0x7F807F80u,
                    ag1 = (argb1 >> 1) & 0x7F807F80u,
                    rb2 = (argb2 << 7) & 0x7F807F80u,
                    ag2 = (argb2 >> 1) & 0x7F807F80u;
    return (((rb1 + rb2) & 0xFF00FF00u) >> 8)
         |  ((ag1 + ag2) & 0xFF00FF00u);
}
The exact same approach (just different numeric constants) can be used to calculate 25%, 12.5%, 6.25%, 3.125%, 1.5625%, 0.78125%, and 0.390625% blends as well.

On 64-bit architectures, expanding the above constants to 64-bit, you can do those blends for two different pairs of color values at roughly the same cost (depending on whether the color values are already packed, or if you need to repack them; so the difference is just some ANDs, ORs, and bit shifts at most).

On 64-bit architectures, an arbitrary ARGB8888 blend only needs two (56×9=64 bit) multiplications, four shifts, six binary ands, three binary ors, one subtraction, and one addition:
Code: [Select]
static inline uint32_t argb8888_blend(const uint32_t      argb0,
                                      const uint32_t      argb1,
                                      const unsigned int  p)
{
    const uint64_t  c0 = (argb0 & 0x00FF00FF) | ((uint64_t)(argb0 & 0xFF00FF00) << 24),
                    c1 = (argb1 & 0x00FF00FF) | ((uint64_t)(argb1 & 0xFF00FF00) << 24),
                    p0 = 256 - p,
                    p1 = p;
    const uint64_t   c = p0*c0 + p1*c1;
    return ((c >> 8) & 0x00FF00FF) | ((c >> 32) & 0xFF00FF00);
}
and you can even apply that to R10G10B10 (30-bit color) by changing the numerical constants.  The "trick" is trivial; second (and fourth) component are shifted up and replaced with enough zeroes to hold the product.

On 32-bit architectures, you can do it with four (24×9=32 bit) multiplications:
Code: [Select]
static inline uint32_t rgb8888_blend(const uint32_t argb0, const uint32_t argb1, const uint32_t p)
{
    const uint32_t  rb0 =  argb0       & 0x00FF00FF,
                    ag0 = (argb0 >> 8) & 0x00FF00FF,
                    rb1 =  argb1       & 0x00FF00FF,
                    ag1 = (argb1 >> 8) & 0x00FF00FF,
                     p0 = 256 - p,
                     p1 = p;
    return (((p0*rb0 + p1*rb1) & 0xFF00FF00u) >> 8)
         |  ((p0*ag0 + p1*ag1) & 0xFF00FF00u);
}
(Total cost being four 24×9=32bit multiplications, three eight-bit right shifts, six binary ANDs, one binary OR, two additions, and one subtraction.)

Obviously, on 8- and 16-bit architectures, you're better off handling each color component separately, as the internal register width is too small for more than one color component, as the intermediate results are 16-bit.

On 32-bit (or 64-bit) architectures, there are a few binary tricks with RGB565 also.  For example, an arbitrary blend (32 steps) between two color values requires just two multiplications (27×6=32 bit), six binary ands (each using one of two 16-bit masks), four shifts, one subtraction, and one addition:
Code: [Select]
static inline uint16_t rgb565_blend(const uint16_t rgb0, const uint16_t rgb1, const uint16_t p)
{
    const uint32_t  c0 = (rgb0 & 0xF81F) | ((uint32_t)(rgb0 & 0x07E0) << 16);
    const uint32_t  c1 = (rgb1 & 0xF81F) | ((uint32_r)(rgb1 & 0x07E0) << 16);
    const uint32_t  p0 = 32 - p,  p1 = p;
    const uint32_t   c = p0*c0 + p1*c1;
    return ((c >> 5) & 0xF81F) | ((c >> 21) & 0x07E0);
}
On 64-bit architectures, you can trivially expand that to blending pairs of color values, for about the same cost.
« Last Edit: August 22, 2019, 03:35:19 am by Nominal Animal »
 

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 6414
  • Country: nz
Re: Neat algorithms
« Reply #119 on: August 25, 2019, 05:53:05 am »
Why do you want to avoid recursion?

The recursion can make the algorithm not deterministic about the use of ram, it might consume too much ram, and I am limited to 4Mbyte on that board, with other stuff that needs ram. I am not even using malloc because I cannot have virtual memory, the CPU does not even have an MMU. Everything must be static and deterministic, with the worst-case known and calculated in advance.

We're talking about B-trees here. With disk-based blocks. What's your block size? How many nodes are in each block? You've probably got a fanout factor of around 100 or so, which means if you have a total data size of 100,000,000 disk blocks (400 GB, say, with 4 KB blocks), then your recursion depth is going to be *four*.  That's not going to run even an Arduino out of stack space.
 

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 6414
  • Country: nz
Re: Neat algorithms
« Reply #120 on: August 25, 2019, 05:53:43 am »
Meh.  Trees (and graphs) are so inherently, obviously, and elegantly recursive that avoiding recursion in   code that implements them is ... cutting off your nose to spite your face.
The algorithms involved are usually well defined and understood wrt their resource usage, and the insistence in some “safety standards” that programmers come up with some less-well-debugged, harder to-understand loop-based equivalent (or use a poorer data structure) is one of the main objections that I have toward such standards.  :-(

Exactly.
 

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 6414
  • Country: nz
Re: Neat algorithms
« Reply #121 on: August 25, 2019, 05:55:10 am »
Isn't the main objection against recursive algorithms, in general, about memory?

The fact that are using the computer stack (to store intermediate results obtained during each recursion step) thus making them slower and memory intensive.
The problem is not that they are memory intensive. Its that unless you take special action their memory use can be unconstrained. A routine that always calls itself a well known range of times has a known maximum memory requirement, and is OK. Many recursive routines call themselves an arbitrary number of times, and will eventually blow the stack up unless special action is taken to constrain the call depth and perhaps declare a failure to compute the true result.

With a B-tree you're going to run out of disk space long before you run out of stack space. They are extremely shallow trees, by design.
 

Offline westfw

  • Super Contributor
  • ***
  • Posts: 4642
  • Country: us
Re: Neat algorithms
« Reply #122 on: August 25, 2019, 06:13:55 am »
OTOH, if the trees are THAT shallow, how much code complexity does the recursion save you?

 

Offline mfro

  • Regular Contributor
  • *
  • Posts: 226
  • Country: de
Re: Neat algorithms
« Reply #123 on: August 25, 2019, 06:57:39 am »
If you have a nice recursive algorithm but fear of overrunning stack, you can easily keep track of your recursion depth with an additional parameter you decrement each call and compare to a (hopefully reasonably choosen) maximum recursion depth (exiting with an error when the threshold has been reached).

This obviously uses extra stack space (and a little performance), so its up to you to to decide if you want to keep it in production code or remove it and call your testing (and additional safety marging due to its removal) good enough to be safe.

 
Beethoven wrote his first symphony in C.
 

Offline brucehoult

  • Super Contributor
  • ***
  • Posts: 6414
  • Country: nz
Re: Neat algorithms
« Reply #124 on: August 25, 2019, 06:59:01 am »
OTOH, if the trees are THAT shallow, how much code complexity does the recursion save you?

Exactly the same amount as any other depth except 0 or 1 :-)
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf