Guides to writing "big" programs

[Infraviolet]

There's loads of guides online for how to learn programming initially, and learn everything you need to so as to write a command line tool with one job, or to write a one-file-plus-several-custom-libraries worth of embedded program... but can anyone recommend any guides on how to take the step from writing simple programs to the sorts of big projects such as a 3d printing slicer, a CAD tool, a video editor, or a networking stack?

Once you get to big programs with lots of functionality, the ability to process complex file in many ways, a GUI with a wide variety of sub-menus, point-and-click and other interaction behaviours, and interfacing with other systems over a complex network (as opposed to simple interfacing over a hadware protocol i2c,uart,spi... to other existing or custom built devices)... how does one go about writing that sort of project?

Does anyone know of good resources on that sort of thing?

Thanks

[brucehoult]

Layers of abstractions.

Look up top-down and bottom-up design.

[golden_labels]

I can only give you the same answer I was giving each time somebody asked me this question in the past 20 years:

Teaching gives us less than flaky foundations. Expertise comes with experience.


Get involved in actually writing something. Starting with maintenance of existing software, aiding in bug fixing, then introducing improvements. That forces you to see how systems are designed and how problems are solved. Involvement may take many forms. A part of junior programmer job. Participating in FOSS projects. Even making changes to software you yourself use, for your own needs. Polish your own skills too, don’t settle for something that merely fits the purpose. Within reasonable limits too: perfectionism is a killer. The goal is to improve, not to never write any program.

There are books that analyze and describe higher-level concepts. But I’m always reluctant to recommend these. They’re great and certainly worth reading. But I myself experienced the deluge of programmers raised on the 1994 “Design Patterns: Elements of Reusable Object-Oriented Software”. What The Gang of Four did is recognizing, classifying, and analyzing existing patterns. What newbies understood is that they’re required to put every existing problem into one of the prefabricated molds from the book.

[brucehoult]

What newbies understood is that they’re required to put every existing problem into one of the prefabricated molds from the book.

Yes, this is bad. It is descriptive, not prescriptive.

Also, the GoF patterns are mostly just papering over the holes in insufficiently expressive programming languages.

https://norvig.com/design-patterns/design-patterns.pdf

[Analog Kid]

Having written some fairly large programs myself (text editor, resource editor), maybe I can give some advice here.

I look at this in a much less abstract way than has been responded to so far. So here are a couple things for a more concrete approach:

if a guy was tasked with writing a big program, here's what he might oughta do at the outset:

• Spend a lot of time before even touching a keyboard sketching, doodling and drawing out some aspects of the program: program flow, at least how you envision it at this early stage, possibly some simple flowcharting (yes! that tool from the 1940s is still a very good one).
  Use paper and pencil here.
• Data structures:
  Since any decent program, and certainly any large one, will necessarily use data structures, try to sketch them out at a top level. They'll inevitably be changed as you start writing code, but at least try to envision the fundamental ones your program will rely on to store its data, define its states, etc.

[eutectique]

Have a look at NASA's Program Model Checking: A Practitioner's Guide

[golden_labels]

I doubly, triply support using paper and pencil!

I didn’t mention it, as to me it’s a basic programming skill. Unrelated to the move from small to big problems. But if you don’t do it yet: absolutely do. Give your brain time to process what you want to create, and use any tools available to visualize the ideas.

If you are stuck with some algorithm, or how some situation progresses, or what are steps in a complicated process: don’t be afraid. Draw that, over and over if needed. Don’t think anybody out there designs things in their mind.

[SiliconWizard]

The key in designing any kind of large "system" is IMO:

- Start with a high-level view and its main functional blocks.
- Define these blocks and how they should interact. Make them as "loosely" coupled as possible unless tight coupling is strictly anticipated either for a safety or obvious performance reason, but don't try "optimizing" in your head too early on.
- Try to avoid circular dependencies between the various high-level blocks as much as possible.
- Think of evolution and maintenance upfront. Avoid strictly ad-hoc approaches at this point: think not about right now, but a few years down the line. "Extensibility" as a goal is a decent stating point.

Once you have that, you can start "toying" with this architecture. For instance, you can write mock-ups of those blocks in the language of your liking (there are systems simulation tools that can be worth investigating at this point) and try to pinpoint any potential shortcoming of the architecture itself. One of such tools is Modelica: https://modelica.org/

Implementation should really come last - when everything above is decently completed, the implementation should be straightforward. Still, there are sometimes some key "components" that can be anticipated as particularly tricky, and so you can prototype those components earlier in the process as a "feasibility" step, to make sure the architecture you have defined is at all feasible, if you have some doubts about some of the required "components" you have identified. You can tag every building block you have defined in the architecture with a "difficulty" note from "trivial" to "not realistically doable", and if you have a team and you're the project leader, do involve your team at this point. Don't do it all on your own, even if you're an expert in the field. If you have to work with others for the project and you don't involve them early, they will resent you to the point of possibly making the project a failure. Of course, if you're a lone designer, you can ignore that.

Just my 2 cents.

[nctnico]

I agree with SiliconWizard.

To add: Writing large pieces of software is system design. Not programming. In today's world agile and continuous delivery are hot and touted to get to a working piece of software sooner but actually these approaches make the life of a system design engineer harder because the architecture needs to cater for future extensions as well. For large systems modelling is important. UML and Hatley–Pirbhai come to mind but there might be more modern variants by now. Hatley–Pirbhai modeling is quite basic and comes down to a process flow (interaction / states) and a data flow (how data is processed) which should match up in a consistent design.

[MarkF]

As a member of 14 engineers who wrote over 700,000 lines of Fortran code over a period of three years:

   The "Guide to writing big programs" is DOCUMENTATION!

There are no short cuts.  You must do the design process.
Breaking the program down into requirements and then divide the
requirements into logical program tasks.  Design walk throughs with your peers.

After your design is broken down into small functions, you get to start coding. 
Expect the design process to take twice as long as the actual coding. 
Don't forget the integration and testing phases.

[chilternview]

• Data structures:
  

Agree, get your data structures down on paper first along with a sketch of the major functional blocks of your code.

Split different blocks into seperate entities (dlls or .so's) and make clean APIs between them.

Think about test right at the start, not after you're written loads of code.

[Analog Kid]

As a member of 14 engineers who wrote over 700,000 lines of Fortran code over a period of three years:

   The "Guide to writing big programs" is DOCUMENTATION!

Yes.
I would go so far as to say that it'd be best if you actually wrote the documentation first, before writing one single line of code.

You can always edit the documentation as the program evolves. But having a somewhat complete manual at the outset can actually help you as a programmer: you can refer to it as if you were working on someone else's code and needed some reference as to what was going on.

(While you're at it you ought to create a second document where you describe internal workings of your program, things the user will never see. It's really easy to create some clever scheme in your software, then completely forget about it when revisiting your own code.)

Now I must admit that I've never actually done this, but I'll have to try it some day.

As an extra benefit, chances are that the final documentation might actually match what the program does and what its dialogs and prompts look like. I was a technical writer in a previous life, and nothing annoyed me more than reading a manual for some software where the text in the manual didn't match what was shown on-screen. Aaaaaargh!

[cfbsoftware]

Check out the books written by Steve McConnell:

https://stevemcconnell.com/books/

[MarkF]

As a member of 14 engineers who wrote over 700,000 lines of Fortran code over a period of three years:

   The "Guide to writing big programs" is DOCUMENTATION!

Yes.
I would go so far as to say that it'd be best if you actually wrote the documentation first, before writing one single line of code.

You can always edit the documentation as the program evolves. But having a somewhat complete manual at the outset can actually help you as a programmer: you can refer to it as if you were working on someone else's code and needed some reference as to what was going on.

(While you're at it you ought to create a second document where you describe internal workings of your program, things the user will never see. It's really easy to create some clever scheme in your software, then completely forget about it when revisiting your own code.)

Now I must admit that I've never actually done this, but I'll have to try it some day.

It's NOT a matter of having or wanting documentation. 
I'm NOT talking about just having a document(s).  I'm talking about the process!

You have to do the design phase.
Identifying requirements and breaking them into small enough pieces to understand and act upon.

If the program is BIG enough you will find that listing requirements and breaking the task into smaller pieces is the ONLY choice you have.  You will also need to divide the program among your coworkers and each will need to understand their part.  Communication will be a big deal.   

For those that think they can cut corners and skip the design phase, I say they have not worked on a big enough program.  Programs so big that it takes a team of people to grasp all the details.  I'm talking multiple year, multi-million dollar, multi-engineer programs.

You will find that large programs will actually require that documentation as a delivery item.
They will also require several customer reviews before you even think about coding.

I would consider "data structures" mentioned above as part of the coding phase. 
Not the design phase.  During the design phase you will not know be able to define what is required by the code and how they interface between the different functions.

[Analog Kid]

I would consider "data structures" mentioned above as part of the coding phase. 
Not the design phase.  During the design phase you will not know be able to define what is required by the code and how they interface between the different functions.

Now that I must totally disagree with.
Any large, complex program, at least in my experience, is going to depend upon and revolve around one or more important data structures.
Therefore those definitely become part of the design phase. Not afterthoughts to be thrown together at coding time.

I agree that you won't be able to completely define the interactions between functions at first, but you probably should at least have a rough idea in your head. Those interactions will of course probably change as you approach and enter the coding phase.

[Infraviolet]

Most of this does ofcourse beg the question of how one even begins to guess what sort of data structures a software package would need to manipulate to do its work. How do you get from the "user facing" functional description of what something should do, across to having an idea where to start planning the underlying structure?

I'm not thinking in terms of a specific project right now, so I can't give examples of various sorts of user interfaces that anyone would then be able to describe in specific detail what the thought process would be from functional use-case to data architecture... but I'm growing more and mroe sure that this is something I am going to need to get an understanding of in future.

Thanks

[MarkF]

The answer is that you can't.

Data structures are part of implementation.
At the requirements and division into functional elements phase of design, you don't know
what computer, what language, or anything about the division into code functions.
You are defining the 'what' you need to do and 'how' you are going to accomplish it.

Pick your standard:  MIL-STD-1644 superseded by MIL-STD-2167.
There is a whole document for 'data structures' and 'interfaces' that parallel the
description document of the code.

[JPortici]

I doubly, triply support using paper and pencil!

I didn’t mention it, as to me it’s a basic programming skill. Unrelated to the move from small to big problems. But if you don’t do it yet: absolutely do. Give your brain time to process what you want to create, and use any tools available to visualize the ideas.

If you are stuck with some algorithm, or how some situation progresses, or what are steps in a complicated process: don’t be afraid. Draw that, over and over if needed. Don’t think anybody out there designs things in their mind.

and that is another reason why i dislike and outright refuse to give in to LLMs to write code.
Either i'm doing something trivial and i just write it instead of having to check the spew of nonsense
Or i'm doing something new, laying out the foundation of a new software, and while writing it (or sketching it out) i have the time to think about it, with the benefit of being in the moment. And using your hands for writing is good for you, for your brain, for understanding.

[tszaboo]

Join a team of programmers.
Argue about irrelevant bullshit.
Have someone try to set up agile and scrum meetings.
Argue on the daily basis now.
Burn out.
Quit.
Go to another company.
Take over legacy code.
Immediately conclude that it's bad, you have to rewrite it.
Argue some more about irrelevant things.
Find out that someone changed all the library/dependency/API or whatever fundamentally, giving you full time work to handle this.
Burn out again.
Repeat.

[Rerouter]

Divide and conquer, the shape of the program will be defined by the team and or the leads approach to the problem

In hardware I'll generally split off specific hardware functions into there own files to keep dealing with any bugs localised to that area,

e.g. you don't want to patch your debouncing logic for an input and suddenly the LCD Uart stops working

with globals, you either go all in, or use them sparingly, e.g. in some embedded enviroments, statically allocating most of the global state of the program is a good way to define it as the global state, ideally you would make your read/write operations to this state clear so you can later search it,

Best description I can give is PCB layout, you group up parts that do a specific function, plumb them together, and then fit it into the big picture, in software its a similar task,

[paulca]

The first thing you do is look for things you don't need to do.  Reuse not reinvent.

GUIs are prime example.  If you look at a basic command line app code size and then compare it to Windows GUI app, the code size is 10-100 times larger.  So GUIs tend to be "auto generated" from IDEs.  You visually lay things out and hit build and it generates all the boiler plate.  In some GUI frameworks the boiler plate is hidden in dynamic runtime code.  But it's still there.  Every single exhausting component, button, menu item, rule line, slider everything needs defined with large amounts of meta dta and properties etc.

So we would pick and select a GUI platform/framework and learn how that works, what that requires.  Windows comes with many of these.  There are cross platform ones like QT and there are web based ones, both HTML/JS style and full GUI applications in a browser ala .Net

The other parts are your database, networking, pretty much everything which is a common component in a tech stack has components and APIs.

They all tend to trade "total flexibility with maximum code overhead" with "opinions and constraints for minimal code overhead."

The big trade though is that their code is already tested and peer reviewed.  Others are using it in production apps.  So it comes with some surety over DIY.

It takes experience to do this though.  You need to have a tool box of components you know about, where to find ones you don't know about.

The modern term used is "Isolation of concerns" rather than divide and conquor.  It decribes what you are trying to achieve through the division, sets clear boundaries and lists clear responsibilities.  When one component becomes responsible for "funtionality X" then NOTHING else does X.

Example:  In one part of your application you need to auto-generate a sensible filename for the user to save their files.  So you start with that code in the routines for saving this file.  This is FINE!  Do not design where there is no pain.  However, later when you now want to save a different file from a different place, but also need a similar filename generated.  Now you can take that code, pull it out and make it stand alone.  Give it an API/Interface and make both use cases employ it to generate file names.  When you are saving a dozen different things and see a bug in filename formatting, you only have one place to go look.

EDIT:  Reading the responses above, there is a mix of 1970s/80s advice and a LOT of embedded advice.  These will not help you with modern applications or systems development.  The code bases are many orders of magnitude apart and the techniques will not transpose as the posters think they will.  It's not 1980s anymore.

[paulca]

OP, you mentioned "Command line" or console apps and you mentioned GUI apps.

In university C++ course this was used as the first example of layering and isolation of concerns employment.

The advice was... write your console app.  However, rather than adding a console interface with text parsers etc.  Instead define an interface on it.  (Just a set of functions it exports for others to use - a contract, almost like a protocol between two components).

Call this your "Backend" or "Service" layer.

Then write your GUI or design it in an IDE.

DO NOT put the two together yet.  Instead for the GUI write a "dummy" impl for your backend service that provides quick test responses that help you do the GUI.

You can also test the "backend" or "service" layer independantly with a "puppet app" pretending to be the GUI calling it.

Then, ideally when your main() runs, it creates the backend, creates the GUI and injects a reference to the backend into the GUI.

Here is a absolute law.  The backend component, knows absolutely nothing about GUIs, widgets, events, and the GUI has no concerns at all about how the backend works.  The only thing the two care about together is the contract of that interface.  This gives you two distinct parallel projects to work on.

As you learn and gain experience you will see this pattern everywhere.  You will see it's done particularly because software needs to be divided up and developed by teams of people, not individuals.  So clear boundaries with mocking and stubbing during dev/test is essential at almost all levels/layers/components.

[nctnico]

As a member of 14 engineers who wrote over 700,000 lines of Fortran code over a period of three years:

   The "Guide to writing big programs" is DOCUMENTATION!

Yes.
I would go so far as to say that it'd be best if you actually wrote the documentation first, before writing one single line of code.

You can always edit the documentation as the program evolves. But having a somewhat complete manual at the outset can actually help you as a programmer: you can refer to it as if you were working on someone else's code and needed some reference as to what was going on.

(While you're at it you ought to create a second document where you describe internal workings of your program, things the user will never see. It's really easy to create some clever scheme in your software, then completely forget about it when revisiting your own code.)

Now I must admit that I've never actually done this, but I'll have to try it some day.

It's NOT a matter of having or wanting documentation. 
I'm NOT talking about just having a document(s).  I'm talking about the process!

You have to do the design phase.
Identifying requirements and breaking them into small enough pieces to understand and act upon.

If the program is BIG enough you will find that listing requirements and breaking the task into smaller pieces is the ONLY choice you have.  You will also need to divide the program among your coworkers and each will need to understand their part.  Communication will be a big deal.   

For those that think they can cut corners and skip the design phase, I say they have not worked on a big enough program.  Programs so big that it takes a team of people to grasp all the details.  I'm talking multiple year, multi-million dollar, multi-engineer programs.

Very true! On some large projects I have developed I spend like 4 to 5 weeks fulltime just on how to structure processes, division into objects and data flows. Not a single line of code! But once a good design is in place, writing code is a real joy because everything falls into place the right way.

[MarkF]

. . .
Very true! On some large projects I have developed I spend like 4 to 5 weeks fulltime just on how to structure processes, division into objects and data flows. Not a single line of code! But once a good design is in place, writing code is a real joy because everything falls into place the right way.

Make more then 1 year in my case.
We were nearly two years in before writing line one of code.

We ended up with 4 three inch binders describing the requirements and
another 15 or so binders for the system design description.  Some misunderstanding
here appears to be that most are unaware of the system design phase.  This phase
has nothing to do with actual code.  It describes how the project works and how
it meets the requirements.

Most of what people are talking about here occurs AFTER these design phases.

The actual breakdown and organization of the software, libraries, and tools used usually
is not a one-to-one mapping between system design items and code items.

This is not about "a team of programmers arguing over a bunch of bullshit and burnout".
How do you organize and layout your software if you can't even remember what it is suppose to do?
This is getting down on paper the big picture, what needs to be done, organizing it, and dividing up the tasks.
When you struggle just to grasp your small little piece of the whole, everyone needs something to refer to if for no other reason than not to forget something.

[paulca]

This is not about "a team of programmers arguing over a bunch of bullshit and burnout".
How do you organize and layout your software if you can't even remember what it is suppose to do?
This is getting down on paper the big picture, what needs to be done, organizing it, and dividing up the tasks.
When you struggle just to grasp your small little piece of the whole, everyone needs something to refer to if for no other reason than not to forget something.

The fact you refer to paper suggests you haven't done this in a while.  The overall situation is as you say, but its fractal.  Large system wide architectures are designed, constrainted and then subsystems and application hang off of it. The difference maybe is that today there is a team doing the architectrual high level and half a dozen other teams doing the components.  Each component itself then breaks down architecturally and maps to "off the shelf" components, patterns, architectures etc.

As an example.  "My" target platform in work is already proscribed.  That means the over-arching system architecture is set already.  The technologies, deployment mechanisms, workflows, CI/CD, databases, middle ware and generate application architecture is already "defeined".  If you want to use something outside of that "Golden architecture" then you need to justify it.

The different between the 1990s and now is that most of the "middleware" and "architectural" level is more rich with off the shelf stuff.  Documentaiton is entirely online.  There are also "funcitonal documentation" aritifacts that serve as both documentation and generation sources.  Stuff like "OpenAPI" specifications of the data flow.  These then get used to generate interfaecs for components, debug UIs like Swagger and even end point documentation for users.

The other main difference is that today the majority of work is not in creating systems, but reworking them, modifying them, replatforming/tooling them, migrating them to new tech-stacks or even new languages.

There used to be a heavy focus on "System's analysis" but the work done there has changed completely in the past 20-30 years.  "System's analysis" used to be about describing messy human business processes in a clear enough way to convert it later to a computer system.

Today there are very, very few fully human business processes left.  So the system analysis has changed quite a bit.  Its usually more about analysing existing messy outdated architecture with all their annoying 1990s BS and brittle rubbsih and modernising them into th ecurrent century.  It's that and redo'ing the software stack done 5 years ago in AWS with all the fancy cloud tools only to have the business reject it and need to redo it because performance sucks or it costs too much to run.