Guides to writing "big" programs

[MarkF]

This is not about "a team of programmers arguing over a bunch of bullshit and burnout".
How do you organize and layout your software if you can't even remember what it is suppose to do?
This is getting down on paper the big picture, what needs to be done, organizing it, and dividing up the tasks.
When you struggle just to grasp your small little piece of the whole, everyone needs something to refer to if for no other reason than not to forget something.

The fact you refer to paper suggests you haven't done this in a while.  The overall situation is as you say, but its fractal.  Large system wide architectures are designed, constrainted and then subsystems and application hang off of it. The difference maybe is that today there is a team doing the architectrual high level and half a dozen other teams doing the components.  Each component itself then breaks down architecturally and maps to "off the shelf" components, patterns, architectures etc.

You're right about that.  Being retired 10 years, I haven't done any of this kind of work since the early 1990's and before.  We were one step from paper tape and punch cards.  Coding in Fortran should have been a clue.  But, the process still holds today no matter modern tools available to aid in the design.  The only disagreement I have is that the work I was doing was leading edge technology.  Nothing to modify or upgrade.  It was one-of-a-kind stuff for the military.  Computers at the time weren't fast enough to perform most of the tasks.  We had to design and build very unique hardware to do the work.  There was even hardware to convert floating point numbers to fixed point to meet processing requirements.

One strives to cut, paste and re-use.  But you have to have something to pull from.

Again, it's not about documentation.  It's the process.  Especially if it's something new.
The documentation is a by-product.

[paulca]

Again, it's not about documentation.  It's the process.  Especially if it's something new.
The documentation is a by-product.

Yea and the nuclear bomb drop in 2026 is the whole LLM agentic assistant thing.  Changes the whole documentation dynamic entirely while oddly keeping it the same.  The change is people use AI to generate documentaiton and then get AI to consume it to do work.  Or at least that's the theory.  I have tested it in hobby land and it's done fairly well in that it's been useful not perfect, but I am still at the "popcorn" stage as to how it pans out on multi-million dollar systems contracts.

[MarkF]

Again, it's not about documentation.  It's the process.  Especially if it's something new.
The documentation is a by-product.

Yea and the nuclear bomb drop in 2026 is the whole LLM agentic assistant thing.  Changes the whole documentation dynamic entirely while oddly keeping it the same.  The change is people use AI to generate documentaiton and then get AI to consume it to do work.  Or at least that's the theory.  I have tested it in hobby land and it's done fairly well in that it's been useful not perfect, but I am still at the "popcorn" stage as to how it pans out on multi-million dollar systems contracts.

We're talking apples and oranges here.

You're talking about implementation.

I'm talking about what comes before.  It's no less relevant today then it was when I did it in the 90's.
It's why multiplication tables are not taught in elementary school.  You just pull up the calculator on your phone.
There will come a time or project that will come where there won't be a choice but to do the work.  Where you won't be able to draw upon what was done before.

[Analog Kid]

The first thing you do is look for things you don't need to do.  Reuse not reinvent.

GUIs are prime example.  If you look at a basic command line app code size and then compare it to Windows GUI app, the code size is 10-100 times larger.  So GUIs tend to be "auto generated" from IDEs.

Not necessarily so.

Small point here, but my experience is writing Windows GUI (Win32) apps in assembly language.
Yes, assembly language.*
I don't want to get into a long discussion about that aspect of it; it's just my preference.
My GUI programs are not much larger than a corresponding command-line app.
Pretty sure most of that bloat is due to monsters like "frameworks".
Even apps made with Visual C are much more obese than mine.

* Disclaimer: I'm certainly not advocating the use of assembly language for real-world production projects. It suits me because I'm a hobbyist and don't have to answer to anyone else with my creations.

[paulca]

I've used asm in production code in a stock exchange, but not in a GUI.  String parsing/message tokenizers etc.  Literally billions of dollars a day when through it for years.

I'm not sure anyone writes for Win32 anymore.  It's a few generations old.  I suppose it's replacement could be called a framework ".Net" and there are been a few rewrites of that.  It's now mostly open source and cross platform.

In the university project I mentioned, I believe the "Engine" component as we called the backend then, was about 150 lines of C++ (A coin change machine, IIRC).  The GUI was created with Borland C++ Builder.  The actual generated files were about 1000 lines.  Remembering that the GUI was required to comply with full GUI conventions.  Proper layouts.  Proper anchors, flow where possible, tab order, tooltips, labels, textbox constraints, validation event handlers, a working close button, a working menu, etc.  In comparison when the CLI had it's own interface it was just a static text menu with options 1, 2, 3, (Q)uit

Borland C++ was a Win32 era IDE IIRC.

That's sort of what I was trying to explain about how "systems" are developed these days.  There are some significant differences to that era today.  We can debate the pros and cons of the lot of it too.  In that era you basically started with nothing.  Win32 was a pretty lightweight API.  So for application development, you basically started fresh each time.   So 'everything' was in your design.  About the only things you got "off the shelf" where databases and operating systems.

Staying in commcerial, non-embedded, mostly business systems where certainly the money is and where most large systems are built/run.

If you needed a 'workflow' to feed data, you needed to design your own queues, state machines, event loops.  If you needed a web server you had to make it from TCPSocket primitives yourself.  In fact, back then, why bother, just open a raw TCP connection and send bespoke binary, right?  Networks are slow, memory and disk are expensive still.  There is no "Internet" in common use.  It's all on you.  With the paper manuals and O'Rielly's books.

Today very little starts from nothing.  Your chances of finding software work starting from a "non-software" system are extremely rare.  O'Rielys books still exist, but a lot of them go out of date so fast you might as well set them on fire when they arrive in the post. 

So when I see "Looks like a queue, smells like a message bus", I see Kafka or JMX, if AWS deployed SQS.  When I "Smell database" I need to ask, "What type?", DocumentDB?  NoSQL DB?  Transactional DB?  Fully RDBMS?  etc.  For each I can lift one off the shelf depending on budget even a full Oracle cluster.  It could even be a big data warehouse like Databricks, Hadoop, et.al.

Those systems you lift off the shelf tend to be opinionated.  They tend to come with a "This is best practices, that's why it's the default way and convention." attitude and your response is either, "Fine, we will bend", or "We will fight you and waste our time."  The good news is, this has led to quite a lot of "harmony".  There are very high advantages to people doing things in similar ways with similar conventions.  "Standardisation" the big one.  Cross platform compat etc.

So once you have picked your infra and tooled up your software layer to integrate with it all, the actual business logic work is pushed into smaller isolated units "hanging off the grand achitecture".

Remember the vast majority of the industry do not use any form of linear methodology like "Water fall".  They are mostly some form of rapid iterative cycle with "Working software" the PRIMARY goal.  Documentation comes about 3rd.  Again not ALL areas can work like that, but I think developer for developer out there its the most common flow.

On the question, "Where did you get 'workflow' from?  How did you see that?  Where did the 10,000ft business process view come from, how did you design that?"

We didn't.  It already existed.  The task was not to "create the digital business process", it was "heres what we already have, heres why we don't like it, here is what are current vision for it is.", fast forward 2 years... "Here's £2mil"

[Infraviolet]

paulca, reply #21, "The advice was... write your console app."

How is this applicable to many types of software though? How would one even imagine what sort of "console" is underlying a CAD program, a word processor, an internet browser...

There are many things among "big" programs for which GUI involvement is so fundamental that the idea of the console without the frontend makes almost no sense.

How does one start understanding one's way out of that kind of problem?

[abeyer]

There are many things among "big" programs for which GUI involvement is so fundamental that the idea of the console without the frontend makes almost no sense.

The goal isn't necessarily to actually have a console interface... you're right it may not be a practical way to interact with many things. The goal is to use it as a way to think about how to decompose and decouple the logic from the details of the interface.

MVC/MVVM approaches are a slightly different way of thinking about the same kind of problem from a somewhate more "gui native" perspective, and might be of interest too.

[ledtester]

Does anyone know of good resources on that sort of thing?

Github.

Find a project that does something similar and see how they organize the code.

Maybe even have Claude help explain it to you.

[paulca]

https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller

Model view controller is one of the "heavier" formal ways of splitting function, data and presentation.

In the simpler model the Controller and View are combined into one GUI and the "model" is your... well model of reality or whatever you are processing.

In bare C, this looks like a module which handle the data and "business" logic while another module does the GUI.

In a word processor the "model" is a "Document model".  It contains all the actual characters as UTF-8, all the control codes, draw macros etc. etc. etc.  When you type, the character is sent to the model.  It updates the document.  It sends a notification to the UI to update.  For very large models like a document, it's likely the model emitts a more contextual signal as to "what it updated".

[golden_labels]

But also don’t get too strict and zealous about the MVC model.

Among design patterns it’s exceptional, as it’s one that a programmer may wish to actively strive for. But it still is a design pattern, a kind of abstraction that tells us how the parts interact. Deviating from the perfect purity is not a mortal sin and the solution should first and foremost fit the problem. There even isn’t any pure form here. The introductory materials to MVC draw an oversimplified picture, but diving into code of different types of programs will tell you that they implemented the model-view-controller distinction and interaction in very different ways.

MVC’s most notable strength is also its Achilles’s heel. In providing a clean dependence path and a set of actions, MVC solves a major issue found in programs with different layouts. That what UI displays gets out of sync from the data or otherwise inconsistent. MVC eliminates this, because UI rendering is always based on the current state of the model, processing it completely. But re-rendering the entire model on each change becomes unfeasible for heavy ones (e.g. a large list, a spreadsheet, or a CAD document).

So do not treat MCV like some fixed, stiff frame you must perfectly fit into. When performance calls for it, deviating from this abstraction is a good choice. It just should not become a habit.

[paulca]

But also don’t get too strict and zealous about the MCV model.

Among design patterns it’s exceptional, as it’s one that a programmer may wish to actively strive for. But it still is a design pattern, a kind of abstraction that tells us how the parts interact. Deviating from the perfect purity is not a mortal sin and the solution should first and foremost fit the problem. There even isn’t any pure form here. The introductory materials to MCV draw an oversimplified picture, but diving into code of different types of programs will tell you that they implemented the model-controller-view distinction and interaction in very different ways.

MCV’s most notable strength is also its Achilles’s heel. In providing a clean dependence path and a set of actions, MCV solves a major issue found in programs with different layouts. That what UI displays gets out of sync from the data or otherwise inconsistent. MCV eliminates this, because UI rendering is always based on the current state of the model, processing it completely. But re-rendering the entire model on each change becomes unfeasible for heavy ones (e.g. a large list, a spreadsheet, or a CAD document).

So do not treat MCV like some fixed, stiff frame you must perfectly fit into. When performance calls for it, deviating from this abstraction is a good choice. It just should not become a habit.

There are some pain points when you consider concurrency and "writes" to the model, especially in web MVC.

You open the page.  That is a controller "GET" operation to get the view.  The model is loaded, serialised and sent to render the view on the browser.  However.  That view may be a form.  The form get pre-filled with the current state.  You edit the state and hit "Save".  Another controller invocation now, POST/PUT and the model is updated and persisted.

Now you have 1000 concurrent users and 5 of them all open the same entity to edit it.  Who wins?

The answers are ugly in most cases.  The choices are basically one of three:

1. That entity is locked by the first one to open the "Edit view".  The other 4 users get a "Termporarily unavailable" error until either that user submits their form or it times out after 5-10minutes.  This is obviously undesirable.  Locks means stale locks, means deadlocks.

2.  All requests are allowed to race.  The last person to hit Save wins.  Request order prevails.

3.  The more common.  A hash of the entity is saved in the form.  When the 2nd user hits "Save", they will get an error that the information is out of date.  Probably reload the form with the current state, erasing their work or just let them spam "Save" again and force overwrite.  The downside with this one is that a lot of backends are "async" to the user scope.  So the "Hash colision" happens long after you have hit Save and you are not aware of the error.  it is silently logged to support and you hope someone fixes it.

In GUI / Desktop MVC concurrency is less "wide" but it does exist in the GUI threading itself.  Your inputs are "Event queue" based and subject to 'ordering' artifacts and events interleaving with running code", "pre-emptive", "re-entrancy" etc.