Menu systems are simple
finite-state machines.
Nowadays, I design mine from the ground up, opposite to how one might think. I start by finding out the knobs/tunables/entries the user will need to adjust the important parameters and operational states, and the different ways of representing those. Then, I create the C data structures (which I keep in Flash/ROM) that should be able to facilitate those, and populate the ones needed for the tunables. Finally, I fill in the entries needed to easily navigate the state machine.
Then, I create a simple C program to run on my desktop/laptop computer, to generate the
actual state transition diagram, by outputting the possible connections using
Graphviz DOT language (it's plain human-readable text), and use the
dot tool to render it into a visible graph.
This tells me if it looks balanced, or if I have long chains of entries (that will annoy users trying to find a specific one) or other problems in the navigation. To fix, I edit the data structures, and rerun. This is an iterative process, and often means changing not just the values, but even the structures themselves. I also like to use things like function pointers for button actions, so the first versions of the structures are often just a basis I start building on.
(I have also done simple domain-specific languages to dynamically generate the constant structures defining the navigation system, but honestly, they don't seem worth the effort to me. The C structures are just as easy to maintain. I have used both individual structures and pointers between entries, and a single array of structures using array indexes instead of pointers, using C99 designated initializers. As an example,
const char x[8] = { [3]=1, [1]=5 }; and
const char x[8] = { 0, 5, 0, 1, 0, 0, 0, 0 }; are exactly equivalent.)
When I believe the structure is acceptable, I create a simulator, either in HTML+JavaScript, or in terminal using e.g. Curses (
ncurses). Then, I find a suitable unwitting
victim test user, and ask them to perform some specific tasks on the simulator, similar to what one trying to use the final device would, and carefully observe their usage patterns with the interface, noting where they go wrong, where they pause to think, and so on, focusing on what does not work well in the user interface (based on mostly the body language and facial expressions of the person; distracting them with a friendly chat makes those easier to observe). The more unwitting
victims test users I can do this with, the better the results will be.
At some point, it is good enough. Perfect is the enemy of good enough, and some users are idiots who expect things to work a specific way so they can avoid thinking and instead rely on their muscle memory. Thus, you must learn where to accept defeat and/or ignore complaints. If you ever tune it so perfectly most users don't even notice the user interface and just intuitively get stuff done without any effort, the downside is that the user interface will be forgotten just as quickly. Everyone remembers a horrible interface, but nobody remembers or even notices a good one, so do expect this to be a thankless task.
At some point during development, depending on how much leeway you have with the physical aspects of the user interface, you should also build a physical example of the UI, or even implement it on the actual target devices' early versions, and test it. Some things like touch screens sound
neat, but if you are doing something oily, dirty, or your hands deep in dough, they suddenly become highly annoying. Same with buttons that are too small, too close, with too high or too low action pressure, and with displays that are perfect in the development environment, but in the actual use case are hard to read/see/clean.
But, full disclosure: I do not do commercial development anymore, just hobbyist stuff, and can focus on the important things instead of making sure I am productive enough for the process to be commercially viable, a profit-making proposition. Do not expect companies' bean counters accept this as a reasonable development method. They'll want to do something different (i.e., cheaper and much faster).