Electronics > Beginners
Debugging a breadboard computer: spurious signals, noise, and voltage weirdness?
PfhorSlayer:
Hello all! This is my first post here, and I'm hoping to make this as detailed as possible with all of the issues I'm facing, the things I've tried, and the type of help I'm looking for. Please forgive me if any of this is either too much (useless) information, or the fact that I'm very much learning as I go starts showing through too much. I'll try to split this post up into a few different parts; if the back story isn't helpful, feel free to skip down to the TL;DR.
I've built Ben Eater's awesome 8-bit breadboard computer, with a few slight modifications. All the pieces work in isolation, but unlike Ben's version (where he uses some sort of black magic or incredible luck), I've had to make a number of modifications to ensure that unused gate inputs are tied (high or low), certain switches are debounced (the ones in his kit and the ones in his videos are different), and things like EEPROM outputs have pull-down resistors to ensure a consistent state while switching addresses. For the most part, though, most of my computer is pretty much the same as what he has designed, circuit-wise. As a brief overview, the computer consists of a clock (with auto and manual modes), a simple ALU that can add and subtract, two 8-bit input registers A and B, a program counter, a cycle counter (each instruction takes 5 cycles), a 4-bit memory address register, 16 bytes of RAM, and an output display module, all connected by an 8-bit bus. The entire computer is built out of 74LS-series ICs, and I've taken photos of my breadboard, which you can see here. There are also schematics for all of the modules on Ben's website.
Once all the parts are connected together, and I'm starting to try running programs with it, I start seeing lots of glitches, errors, and other problems that I'm not knowledgable or skilled enough (yet!) to resolve myself. I'm a software guy, and this is my first real (since I was a small child playing with those 300-in-1 kits) electronics project.
The main issue I'm trying to resolve at this time is that the memory address register, which controls the address that is passed to the RAM chips, occasionally resets itself to zero while stepping through a certain instruction cycle. It does not happen every time, and occurs much less often when the register is hooked up to an oscilloscope, but when it does happen it seems to be pretty consistent, always on the same instruction and cycle. It happens on the falling edge of the clock pulse, and the symptoms look as follows: first, the rising edge of the clock pulse pulls the program counter into the memory address register (which I can see happen via the blinkenlights); next, on the falling edge of the clock pulse, the address register is cleared or reset to zero.
I have probed all parts of the 74LS173 register chip, and some screenshots of the results are in the photo album.
There is a clear signal, activated via a push-button, that resets the computer's registers and state back to zero. It seems that sometimes, this signal is raised to around 2V when this glitch happens, which resets the register (but ONLY this register, not any of the others connected to the same signal!), despite the button not being pressed. At other times when the glitch happens, I see the Vcc pin for the register going up to ~6.3V, despite the fact that the computer is running at 5V.
Here's my main question: how do I go about debugging this kind of thing? When I see a weird signal, or Vcc doing something odd, what kind of steps should I take to try and isolate the problem? It's much easier when the domain is smaller, like when trying to debug an issue with a single module, but when the issue only happens when the entire computer is connected together, I'm presented with such a large problem space that I don't even know where to begin.
What kind of debugging methods should I employ in trying to fix these glitches? Are there some steps I can take to reduce the problem space? At the moment (as a software guy), this feels akin to me trying to debug an optimized executable, without symbols, by stepping through instruction-by-instruction. It could be done, but it would be such a monumental undertaking that it would take forever without additional tools. That's what I'm looking for here, additional tools I can use to help me debug these issues.
Beyond what's shown in the original schematics, here's what I've already done:
* Added pull-up resistors to all unused gate inputs; in many cases, I've wired all the unused inputs together, and used a single 1K or 10K resistor to pull the whole set to Vcc.
* Isolated the edge-detection RC circuit on the clock that's controlling the RAM write from the rest of the clock signal.
* Changed the auto/manual clock switching 555 timer to be using a monostable configuration, as the switch I have bounces and was causing extra clock cycles if switched at the wrong time. Similar to how the single-step manual clock switch works, the 555 timer remains active while the trigger is high (and is also debounced).
* Placed a lot of decoupling caps across the power rails, and added a huge 1000nF cap where the power comes in from the wall plug.[li]Added pull-up/down resistors to many outputs where I needed to make sure that the value is never floating. The EEPROM outputs, for example, are all tied low to make sure that they don't float while the EEPROMs switch addresses.
[/li][/list]
Are there other common things I should try? The computer is powered by a 5V, 2A wall plug, and I'm getting roughly 4.8V on all the power rails on each breadboard. This should be enough to handle everything, as far as I can tell.
Thank you all for any help or insight you can give me! If (when) additional info is needed, please let me know!
TL;DR:
I need some guidance on how to go about debugging issues in a large, complex system where all the individual parts seem to work great in isolation.
jhpadjustable:
Welcome! Just a note, local custom encourages you to upload your photos to the forum itself, rather than linking. If imgur goes belly-up tomorrow, then what?
--- Quote ---There is a clear signal, activated via a push-button, that resets the computer's registers and state back to zero. It seems that sometimes, this signal is raised to around 2V when this glitch happens, which resets the register (but ONLY this register, not any of the others connected to the same signal!), despite the button not being pressed. At other times when the glitch happens, I see the Vcc pin for the register going up to ~6.3V, despite the fact that the computer is running at 5V.
--- End quote ---
Yikes. Power distribution is up to 100% of your problem. I suspect the clear signal is getting pulled high by that power spike, whether by input leakage or otherwise, and doing what it was meant to do. I took a quick look at your photos and here are my notes:
1. Real wires are inductive. Each IC package needs to be decoupled from the rail and the rail's inductance. The way you're decoupling that chip, there could be more than two breadboard lengths worth of inductance between the Vcc of the cap and the Vcc of the chip, counteracting the effect of the decoupling cap, never mind the distance between the chip and the power supply, which might be the cause of that current spike. Ideally, the cap should be right at the power pins, or as close as you can reasonably manage, with leads as short as feasible. The sketch below illustrates what should be an effective compromise:
It wouldn't hurt to keep a uF or more of electrolytic capacitance every 10-15cm or so along each rail, but see 1a below.
1a. Real wires are also resistive. Observe that the top right power rail plug-bar is carrying the entire current of your system, with its inductance and resistance introducing predictable effects on stability of the rest of the circuit's power. I'm only about 51% joking when I suggest building bus bars out of thicker gauge wire (14AWG/1.5mm diameter or so) and soldering stub wires onto them for plugging into the rails.
2. 74LS series logic specifies 4.75V minimum Vcc, which must be observed at all times, including transients, to guarantee correct operation. Chargers aren't especially well-regulated because they don't need to be. Try to find a beefier power supply, if you can. Or, choose a logic family more tolerant to lower supply voltages (74HC?).
3. TTL output currents are not symmetrical. Especially note that 74LS is only guaranteed to supply 0.4mA when driving high (but 8mA when driving low). A 1k pulldown would dominate that, slowing or stopping the rise into the logic-high voltage range. Use a 4.7k pullup instead, and as a general rule, prefer pullups to pulldowns.
Tools... you've got a scope and you've used it effectively to find the problem. :-+ Before you make the changes above, you should probe all around the power rails as you reproduce that glitch, so you can get some idea of how wide the effects can be spread. You might also find a logic analyzer to be helpful, maybe not for this sort of problem, but for others that may come to haunt you down the line, especially where you want to record more data points than the scope will allow then review them off-line at your leisure. You don't need an especially expensive one. One of the $20 "Saleae Logic" clones running the FOSS fx2lafw firmware + the FOSS sigrok software would be quite helpful.
You may be aware that gdb's step and next commands take an optional numeric parameter. You would probably enjoy having something similar for your build so that you can advance n cycles at will. A counted clock would be fairly easy to build out of a few counters (74LS163), a flip-flop or three (74LS74?), DIP switches to set the cycle count, an AND gate to gate the clock, and a debounced start signal. If you're not too purist about tools, it would be even easier to build with nearly any microcontroller and its built-in timers.
tggzzz:
That's a well constructed first post :)
jhpadjustable's points are valid.
I'll add that you need to consider the (lack of) reproducability and longevity of connections made in solderless breadboards. What works today might not work tomorrow after you have knocked it :(
Other construction techniques are wirewrap (expensive nowadays, good enough for LSTTL), or "manhattan" (good for higher frequencies). But if you want to continue using the computer, consider using a CAD package for the schematic and PCB layout, and getting a PCB made.
PfhorSlayer:
Thank you very much, jhpadjustable! I probed around a bit more, and found some areas where Vcc got dangerously close to 4.75V, and a couple places where it dropped under, during the spike. Power is definitely the issue!
With your points in mind, I added a second twisted-pair of wires coming from the power plug (and soldered together where they meet at the plug, and crammed in there as tight as possible), and hooked those to the breadboard near where the original two were. I noticed with the single set of wires, that when moving them around slightly while attached to the breadboard, the LEDs would flicker slightly. Definitely not good, but adding the second pair of wires seems to have helped out there quite a bit. I've ordered some header pins, and when they arrive I'll solder both pairs of wires to them and plug it in to the board. Hopefully, that'll be good enough for what I'm doing here!
And yep, it was the lack of decoupling caps directly across the ICs that was causing the resetting issue! I originally figured that just putting them on the rails would be good enough, but as you mentioned, if it's placed next to where the chip's Vcc is attached, the ground lead needs to go all the way down the length of the rail, across the jumper wires to the bottom rail, and back along the length of THAT rail to get back to the ground pin! That's crazy long, and it's now quite obvious why it's not working correctly. I added a small ceramic similar to what you drew out, and sure enough, the resetting problem went away entirely! I'm going to repeat that process for all the other register chips on the board, and as many of the others as I have caps on me to cover. Thank you!
As far as the pull-up/pull-down, the only reason I pulled the EEPROM outputs down was because I wanted the "default" state to be low for them, as their signals control the rest of the computer and having them default to high would lead to modules being activated when they should not be, or multiple things trying to output to the bus at the same time, which would definitely not be good!
tggzzz: putting it on a proper PCB is definitely the goal, but obviously I need to be sure everything actually does what it's supposed to before I start expanding it (and making it permanent)! :)
tggzzz:
--- Quote from: PfhorSlayer on November 28, 2019, 09:47:55 am ---And yep, it was the lack of decoupling caps directly across the ICs that was causing the resetting issue! I originally figured that just putting them on the rails would be good enough, but as you mentioned, if it's placed next to where the chip's Vcc is attached, the ground lead needs to go all the way down the length of the rail, across the jumper wires to the bottom rail, and back along the length of THAT rail to get back to the ground pin! That's crazy long, and it's now quite obvious why it's not working correctly. I added a small ceramic similar to what you drew out, and sure enough, the resetting problem went away entirely! I'm going to repeat that process for all the other register chips on the board, and as many of the others as I have caps on me to cover. Thank you!
--- End quote ---
You are now coming to the understanding that you are building an analogue circuit, albeit one where chip inputs can interpret the analogue input voltages as digital signals. This topic is called "signal integrity", and is very important.
Non-registered outputs can cause problems w.r.t. tsu setup times, particularly if they drive a clock input.
It is good practice to put decoupling capacitors on all ICs leads, and to ensure you have a very good ground system. Ideally it is a solid ground plane, but for slow stuff like LSTTL you can get away with grids of ground connections.
The key calculations you should do are to realise that for an inductor the induced voltage (a.k.a. back EMF) is Vi = L di/dt. Rule of thumb is 1mm of wire = 1nH, but what is di/dt value? Assume dt is the output transition time (not propagation delay), say 5ns for LSTTL. To work out di, start by calculating
* the difference in output current when driving a logic 1 or 0 (1.6mA for LSTTL IIRC?), plus
* the current required to charge an input's capacitance. This is i = C dv/dt, where dv is 2V for LSTTL (2.4-0.4V)and multiply that by the total number inputs connected to the IC's outputs. Strictly you should also include an allowance for switching spikes inside a TTL IC, but the above calculation is sufficient to make the point.
Compare that Vi "noise" with the 0.4V headroom on LSTTL inputs in the low state; you will be surprised.
--- Quote ---tggzzz: putting it on a proper PCB is definitely the goal, but obviously I need to be sure everything actually does what it's supposed to before I start expanding it (and making it permanent)! :)
--- End quote ---
There's a danger of spending too much time associated with debugging problems inherent in solderless breadboard constructions. Manhattan techniques often have a "tile" for a subcircuit, and when one is debugged it is connected to debugged other tiles. See https://entertaininghacks.wordpress.com/2020/07/22/prototyping-circuits-easy-cheap-fast-reliable-techniques/ for some examples. They inherently have good ground planes, see above.
Navigation
[0] Message Index
[#] Next page
Go to full version