FPGA VGA Controller for 8-bit computer

[nockieboy]

Do you have a good image of the dev board so I can see this layout.
You may get lucky and only have to razor cut 8 traces depending on layout.

Here's a couple of images.

[BrianHG]

Do you have a good image of the dev board so I can see this layout.
You may get lucky and only have to razor cut 8 traces depending on layout.

Here's a couple of images.

Everything is fairly straight.
You should have no problems with 480p.  To attempt 720p, you would just need to remove the ram chip or cut the 8 traces to the ram.  Though, if 480p works, I wouldn't bother.

Just do as I said in the last post and start by making a PCB with the amp chip & a 3.3v regulator being fed from the 5v on the connector.

for each data pair, use this wiring:

   VCCIO
        |
      50Ohm 0603
        |
IOp ---> 0603 .1uf ---> NXPINp
IOn ---> 0603 .1uf ---> NXPINn
        |
      50Ohm 0603
        |
    VCCIO

(You may swap the n&p if it will keep the routing straight)

I checked the datasheets and differential driver & receiver's capabilities for both chips.  The above circuit covers the worst case scenario.
Most likely, you will test with the 0.1uf changed for jumpers or 50Ohm or 25Ohm resistors ignoring the other 2 50Ohm terminators to VCCIO as in DC mode, the NXP chip has built in 50ohm pullup terminators to it's own internal 1.8v reference.  The NPX chip is specified to support differential inputs with voltage swings anywhere from 0.3v through biased to 1.8v.

The 3.3v regulator will power the NXP amp.

The 3.3v VCCIO coming from the dev-board's connector will be used to power the VCCIO 50 ohm terminators.
When powering the CIV to 2.5v (make sure you purchase a 2.5v regulator for your dev board), this 3.3v will change with it.  This will be in proper spec for the CV differential output in  Differential SSTL-2 Class I & II & the NXP internal 50 Ohm terminator will be almost in spec with jumpers in place of the 0.1uf caps.  With 25ohm series resistors in place of the 0.1uf caps, the pull-up load of CIV's differential outputs will be completely within spec.  (SSTL-2 requires a 50ohm to 1.25v termination.  The NXP has a 50ohm to 1.8v termination.  The series 25 ohm lowers the pullup current on the CIV output driver to appear very close to as if a 50ohm was tied to 1.25v since now it will look like a 75 ohm tied to 1.8v)

[asmi]

Here's attempt No.1 for the DVI output card.  Have done what I can with it, but there's probably loads of things wrong with the power layout. 

The ground connections between HDMI lines at connector are not good. Get rid of that trace, and instead add a via to the ground right next to each ground contact. And try getting into the habit of using a via with the shortest possible trace to it any time you need a connection to the ground (or power in case of 4+ layer boards). Long power/ground traces render decoupling capacitors useless as current can't get to device fast enough due to trace inductance.
Also try to make all HDMI lanes the same length. You've got a ton of space on a board, so that shouldn't be a problem at all. Unbroken ground plane under HDMI traces is a must.
Next, add a low dropout Shottky diode between connector's +5V line and the rest of the board to prevent backpowering of your board from the HDMI sink - it should let the current go from your board into connector, but should block reverse direction. As per HDMI spec the max. current through 5V line should not exceed 50 mA, so you don't have to add anything beefy.
Finally, it's usually not the greatest idea to connect a metal shroud of connector directly to the ground, instead do it via fat 1M resistor (I usually use 1206 size) and a high-voltage 1nF capacitor (again 1206 size) directly across that resistor. This will allow static charge to safely discharge into the ground via resistor, while shorting HF signals directly to the ground via capacitor.

[BrianHG]

Yup, the bottom of that PCB should be all GND filled.
And each HDMI GND pin should have it's own VIA to that GND layer.

Make sure there are 100nf caps for each supply pin on the NXP IC and at least 3 for the 50 ohm pullup terminator resistors.

Also add 1 10uf for each supply rail.

[asmi]

Also - I would consider going for 0.8 mm PCB instead of standard 1.6 mm if possible, that will bring HDMI traces' impedance closer to their spec of 50 Ohm single ended/100 Ohm differential impedance. Remember - the thinner the isolation between trace and a reference plane, the lower the impedance.

[nockieboy]

Latest update on the DVI card.  Think I've taken into account all the advice above, with the exception of HDMI signal length matching.  Let me know if the power trace positioning on the bottom layer is okay. 

[nockieboy]

Everything is fairly straight.
You should have no problems with 480p.  To attempt 720p, you would just need to remove the ram chip or cut the 8 traces to the ram.  Though, if 480p works, I wouldn't bother.

That's great then - I don't want to be cutting traces on the dev board if I don't have to.  Never know if I might need to use the dev board to test out some DRAM interface... 

Also, just had a quick look at the schematic for the dev board to see what negative affects swapping the 3.3V regulator for a 2.5V one would make - can't see any issues to my untrained eye, other than perhaps the 50MHz crystal getting 2.5V instead of 3.3V, but that shouldn't be an issue for the FPGA, hopefully.  I actually have some 2.5V AMS1117s somewhere, might swap that regulator this weekend and test to make sure it's all still working okay (with the exception of anything on the IOs that only works with 3.3V, obviously).

OK guys, since you liked my previous one, I have another proposal Whenever I start working with new (for me) FPGA, I always design and build a simple breakout devboard with the bare minimum components to run the FPGA, and some memory (as you usually can't connect it via connectors) and just a couple of LEDs and buttons.

This is a good idea, asmi, but I'm really wanting to push forward on the GPU project.  I'll give this suggestion some serious thought once I've got a run of GPU cards and can start on the HDL side for the Cyclone V.  Seems like a good way to practice some BGA soldering - if I can make the dev cards small enough and cheap enough, it won't hurt to use them to get confident in BGA soldering. 

[BrianHG]

Add these 4 Differential LVDS compliant 100 ohm resistors.
See photo.
Note that the Cyclone's differential IO's have software controlled series and parallel resistors and support many differential SSTL & LVDS modes.  The NXP seemed to be designed to take almost anything, even after a long cable and automatically amplify the signal into a DVI/HDMI TMDS compliant signal to drive through another cable.

With our test board, we will pick the cleanest functional solution, though, I do admit it would be a pluss if you had access to a 500MHz scope...

[nockieboy]

Added the terminating resistors - is there a particular part I should be looking for on Mouser for these, or will any 0603 100 ohm resistors do?

With our test board, we will pick the cleanest functional solution, though, I do admit it would be a pluss if you had access to a 500MHz scope...

Yeah, wish I had one of those too.  Or the money for one.  They seem to range from $350 upwards on eBay.  My oscilloscope is only 20MHz by the way, not 25. 

[BrianHG]

Added the terminating resistors - is there a particular part I should be looking for on Mouser for these, or will any 0603 100 ohm resistors do?

Any 0603 will do.  Also, for the 50ohm, 51ohm will also do if there is no stock.

[nockieboy]

Any 0603 will do.  Also, for the 50ohm, 51ohm will also do if there is no stock.

Marvellous.  All good then?  Happy with the design?  I'm going to be ordering parts for the main GPU card at the same time - minus the TFP410s and ADV7125s.  Any ideas on DDR RAM?  Would like to order that at the same time to save on shipping.

[asmi]

Marvellous.  All good then?  Happy with the design? 

Can you pls post latest schematics?

I'm going to be ordering parts for the main GPU card at the same time - minus the TFP410s and ADV7125s.  Any ideas on DDR RAM?  Would like to order that at the same time to save on shipping.

It depends if you want DDR2 or DDR3. Latter has more capacity for the same price, and it's always nice to have more memory, but on the downside it's going to require additional DC-DC converter because DDR3 is powered by 1.5 V (DDR3) or 1.35 V (DDR3L). The latter one is a bit more picky about layout, but it consumes less power, and even DDR3 is more power-efficient than DDR2. That said, I've personally never had any problems with DDR3L implementation, so maybe that "picky-ness" only manifests at much higher frequencies than I was running at, and lower power means memory device stays cooler. Also all DDR3L parts are 100% backwards compatible with DDR3 and can be ran at 1.5 V as well as 1.35 V.
For DDR2 I recommend MT47H128M16RT part (it's 128Mx16=256 Mbytes part, the maximum capacity available for DDR2), for DDR3 - MT41K256M16TW - this one is 256Mx16=512 Mbytes part. There are larger capacity DDR3 devices out there (8Gbit parts or even dual-die 16 Gbit ones), but they are disproportionally more expensive in my opinion.
Also keep in mind that you can always install smaller capacity memory device on a board designed for larger one - higher address lines just won't be used. So I always recommend to do a layout for the largest capacity possible, but use what you think is reasonable. This way if you will require larger capacity down the road, you can just install larger device onto the same PCB without making any changes to the board.

[asmi]

This is a good idea, asmi, but I'm really wanting to push forward on the GPU project.  I'll give this suggestion some serious thought once I've got a run of GPU cards and can start on the HDL side for the Cyclone V.  Seems like a good way to practice some BGA soldering - if I can make the dev cards small enough and cheap enough, it won't hurt to use them to get confident in BGA soldering. 

That's OK and of course it's your call. The reason I usually begin with such board is very simple - since nobody helps me with schematics, I need to make sure I did all basic connections right (things like PLL connection, flash memory, JTAG, etc - things you need to get right for FPGA to even work at all). And since the very first project on a new (for me) FPGA carries much higher than usual risk, I'd rather retire it on simpler board first, as it would be cheaper to respin it if I screw something up.

That said, you always need to be prepared for the fact that you will very likely need to do a one or two revisions before you get everything working as it should. This is unfortunate, but this has been my experience - in most my devboards the first revision is screwed up beyond repair as I got some major thing wrong, second revision kind of works (found more issues, but they are usually fixable via some bodging and cutting), and only third revision is perfrect and works 100% how it was intended. By now I have a dedicated "box of shame" where I store all my screwed up revision A boards as a reminder not to rush into production and instead check everyting many many times over. There are very things which are worse than finding a major flaw in your board just as it went into manufacturing and you already know it's going to be a fresh addition to the "box of shame" even though it doesn't even fully exist yet

[BrianHG]

RAM table:

Code: [Select]

Example possibilities :

DDR2 - W9751G8NB-25     =  64mx8  - 1.8v - 60 ball BGA  2.80$
       W9751G6NB-25     =  32mx16 - 1.8v - 84 ball BGA  2.08$
DDR3 - W631GG8MB-12     = 128mx8  - 1.5v - 78 ball BGA  1.76$
       W631GG6MB-11     =  64mx16 - 1.5v - 96 ball BGA  1.00$

8 bit ram IO requires:
Requires (8bit data + DQSn/p bank + PLLoutn/p) + address + banks + odt/cs/ras/cas/we
         (        8 + 2 +           2        ) + 14      + 2     + 5
----------------------------------------------------------------------
                12 (1 bank with DQ & DQS pair) + 21 regular IOs at ram IO voltage


16 bit ram IO requires:
Requires 2DQbank(16bit data + 2xDQSn/p bank + 1xPLLoutn/p) + address + banks + odt/cs/ras/cas/we
         2DQbank(        16 + 4             + 2          ) + 14      + 2     + 5
----------------------------------------------------------------------
                 22 (2 banks with DQ & DQS pair) + 21 regular IOs at ram IO voltage

Add 3-4 more IOs for DDR3.
Check the data-sheets.  If you can run the DDR3 as slow as 200MHz, 400mtps, then go for DDR3 for sure.
If you can achieve 16 bit, then go for 16bit, though, you will need to make at least 3-4 IO banks at 1.5v.
You will need to change the 1.8v/1.5v linear regulator to a switcher just in case.  DDR2/3 do draw more power.

Other ram manufacturers will work.
I have never done a DDR3 design before, only DDR2.
(Been retired from electronics for 10 years now.)

[asmi]

Check the data-sheets.  If you can run the DDR3 as slow as 200MHz, 400mtps, then go for DDR3 for sure.

Didn't you say that CV can run DDR3 at 325 MHz (or something like that)? I think the minimum for DDR3 with DLL on (which is a normal mode of operation for DDR3) is 300 MHz.
This External Memory Interface Spec Estimator says that hard memory controller (which is present in all CV devices, some have more than one such block) can run DDR3 and DDR3L at 333 MHz in CV grade C8.

If you can achieve 16 bit, then go for 16bit, though, you will need to make at least 3-4 IO banks at 1.5v.

For sure go for x16 - that's double the bandwidth essentially for free as x16 memory chips are readily available.

You will need to change the 1.8v/1.5v linear regulator to a switcher just in case.  DDR2/3 do draw more power.

Agreed. Switchers also have better transient response, which is beneficial for memory.

Other ram manufacturers will work.

I only ever used Micron devices because as far as I can tell, it's the only manufacturer that has published full HDL simulation models for their memory devices. And the latter is a big thing if you're trying to implement a memory controller yourself.

I have never done a DDR3 design before, only DDR2.
(Been retired from electronics for 10 years now.)

I have done both DDR2 and DDR3L recently, and there is no substantial difference between them as far as layout does - at least when operating at lower frequencies (400 MHz and below). If you need to run faster, than you will need to do 40 Ohm traces instead of 50 Ohm ones, and it's significantly more tricky without using some expensive prepreg materials. The controller is different though due to the requirement of doing the write levelling for fly-by routing, but since I only ever used pre-made controllers, I didn't have to worry about these differences at all. And I don't think write levelling is really required for a single x16 memory device - it is more for cases when you have multiple memory devices (like using DIMM module).

[BrianHG]

Check the data-sheets.  If you can run the DDR3 as slow as 200MHz, 400mtps, then go for DDR3 for sure.

Didn't you say that CV can run DDR3 at 325 MHz (or something like that)? I think the minimum for DDR3 with DLL on (which is a normal mode of operation for DDR3) is 300 MHz.
This External Memory Interface Spec Estimator says that hard memory controller (which is present in all CV devices, some have more than one such block) can run DDR3 and DDR3L at 333 MHz in CV grade C8.

303MHz/606mtps software mode, half rate 2:1 controller.  400MHz/800mtps with hard controller which even operates in full-rate mode.   We will be running the C8 as if it were a C7.  This is not a problem as all I wanted was to match the ram clock with a multiple of the core.  If we switch the core from 250MHz to 200MHz, running the ram at 400MHz would be great.  Our geometry processor will loose a little speed.  Overclock to 432MHz would be great since that's 27Mhz * 16.  (unless we pipeline the heck out of the geo unit, we can then jump it from 100Mhz to 200Mhz.  This could be something for a later date.)

The hard controller demands that the DQ IOs and DQS and CLK out are on specific pins (this I am sure) and the address/command lines are in an adjacent bank with direct connection to the same PLL (this may be more flexible).  But, Nockieboy should read the Intel's External Memory Interface Controller's user guide to make sure we get everything right.  1 mistake means the 400MHz gets cut down to 303MHz and I would not like to make a design which has multiple clock domains.  (I do not consider 2:1 & 1:2 different clock domains as our core is wired it like a DDR inside a second DDR just fattening the data bus, or offering 2 simultaneous access ports per 1 system clock as we are currently doing)

If the NXP chip does what it claims, it should also be able to be driven by a differential 1.5v LVDS IOs.  This just helps as each IO bank has 16 IOs and using half for HDMI and half for DDR addressees may help on available IO resources.
(We can simulate driving the NXP with 1.5v LVDS on the current HDMI test board thanks to my last addition of 4x 100ohm balanced terminator resistors.)

[asmi]

The hard controller demands that the DQ IOs and DQS and CLK out are on specific pins (this I am sure) and the address/command lines are in an adjacent bank with direct connection to the same PLL (this may be more flexible).  But, Nockieboy should read the Intel's External Memory Interface Controller's user guide to make sure we get everything right.  1 mistake means the 400MHz gets cut down to 303MHz

Why don't you create a design first in Quartus, including pinout, and do a complete place & route and timing analysis before you commit to manufacturing the board? This way you can be sure that the pinout you've chosen will actually work. Xilinx memory generator always generates example design along with memory controller, such that it can be actually implemented in hardware and used to check memory operation. As far as I understand, Antel memory IP does the same thing, or is it?
Whenever I work on something with high-speed IO, I always do a design first - even if incomplete, but enough to see IO part working at full speed. This saved quite a bit of revisions from being duds due to bad IO choices.

and I would not like to make a design which has multiple clock domains.

Well if you implement a cache system like I suggested few posts back, the memory frequency will not matter (as long as there is enough bandwidth of course), because working with DDR will be decoupled from the rest of design, which will work with few BRAMs, while DDR side will use it's own BRAM port.

[BrianHG]

Other ram manufacturers will work.

I only ever used Micron devices because as far as I can tell, it's the only manufacturer that has published full HDL simulation models for their memory devices. And the latter is a big thing if you're trying to implement a memory controller yourself.

Agreed, this one is the cheapest (Automotive DDR3): MT41K64M16TW-107, but you are paying a premium of 2x at $1.98 each.  The 128mx8 version -> MT41K128M8DA-107 <- is cheaper.  This would leave a lot of free IOs and at 800mtps, even at 8 bit, the throughput still wouldn't be too bad.

Though you would need to run the device at 1.35v, or up to 1.45v.  Otherwise you would need 2 regulators to get the 1.5v and 1.35v for ram core.

For normal DDR3, 1.5v, use this one -> MT41J128M16JT-125K <-, 128mx16 @ 2$.
This is a 1.6GHz part, can it be run at 300MHz?
256mx8 -> MT41J256M8

I wonder if there are any reduced latency DDR3 rams.  Cut down the RAS-CAS-DATA cycle.

[BrianHG]

The hard controller demands that the DQ IOs and DQS and CLK out are on specific pins (this I am sure) and the address/command lines are in an adjacent bank with direct connection to the same PLL (this may be more flexible).  But, Nockieboy should read the Intel's External Memory Interface Controller's user guide to make sure we get everything right.  1 mistake means the 400MHz gets cut down to 303MHz

Why don't you create a design first in Quartus, including pinout, and do a complete place & route and timing analysis before you commit to manufacturing the board? This way you can be sure that the pinout you've chosen will actually work. Xilinx memory generator always generates example design along with memory controller, such that it can be actually implemented in hardware and used to check memory operation. As far as I understand, Antel memory IP does the same thing, or is it?
Whenever I work on something with high-speed IO, I always do a design first - even if incomplete, but enough to see IO part working at full speed. This saved quite a bit of revisions from being duds due to bad IO choices.

I was getting to the point of asking Nockieboy to do at minimum add a DDR3 controller to the current design with a minimal bridge to the current code (IE as Z80 read/write ram access) just so he can confirm full timing closure in Quartus with his chosen IO pin connections.

[BrianHG]

and I would not like to make a design which has multiple clock domains.

Well if you implement a cache system like I suggested few posts back, the memory frequency will not matter (as long as there is enough bandwidth of course), because working with DDR will be decoupled from the rest of design, which will work with few BRAMs, while DDR side will use it's own BRAM port.

I want Nockieboy to code this, not me...
And it's has nothing to do with any cache system.
The new CV has plenty of spare ram blocks for multiple dual port, true dual clock.

[nockieboy]

RAM table:

Code: [Select]

Example possibilities :

DDR2 - W9751G8NB-25     =  64mx8  - 1.8v - 60 ball BGA  2.80$
       W9751G6NB-25     =  32mx16 - 1.8v - 84 ball BGA  2.08$
DDR3 - W631GG8MB-12     = 128mx8  - 1.5v - 78 ball BGA  1.76$
       W631GG6MB-11     =  64mx16 - 1.5v - 96 ball BGA  1.00$

8 bit ram IO requires:
Requires (8bit data + DQSn/p bank + PLLoutn/p) + address + banks + odt/cs/ras/cas/we
         (        8 + 2 +           2        ) + 14      + 2     + 5
----------------------------------------------------------------------
                12 (1 bank with DQ & DQS pair) + 21 regular IOs at ram IO voltage


16 bit ram IO requires:
Requires 2DQbank(16bit data + 2xDQSn/p bank + 1xPLLoutn/p) + address + banks + odt/cs/ras/cas/we
         2DQbank(        16 + 4             + 2          ) + 14      + 2     + 5
----------------------------------------------------------------------
                 22 (2 banks with DQ & DQS pair) + 21 regular IOs at ram IO voltage

Add 3-4 more IOs for DDR3.
Check the data-sheets.  If you can run the DDR3 as slow as 200MHz, 400mtps, then go for DDR3 for sure.
If you can achieve 16 bit, then go for 16bit, though, you will need to make at least 3-4 IO banks at 1.5v.
You will need to change the 1.8v/1.5v linear regulator to a switcher just in case.  DDR2/3 do draw more power.

Other ram manufacturers will work.
I have never done a DDR3 design before, only DDR2.
(Been retired from electronics for 10 years now.)

I'm having trouble sourcing those parts you've suggested from Mouser.  I'm going through a legitimate retailer this time around rather than AliExpress for my parts and Mouser is my preference.  Would this be any good for DDR3 - MT41K128M8DA?

Agreed, this one is the cheapest (Automotive DDR3): MT41K64M16TW-107, but you are paying a premium of 2x at $1.98 each.  The 128mx8 version -> MT41K128M8DA-107 <- is cheaper.  This would leave a lot of free IOs and at 800mtps, even at 8 bit, the throughput still wouldn't be too bad.

Getting lost in choices, here.  So we're definitely going DDR3, yes?

I like the look of this one - less signals to route = less headache for me.    And it's in stock at Mouser. 

Though you would need to run the device at 1.35v, or up to 1.45v.  Otherwise you would need 2 regulators to get the 1.5v and 1.35v for ram core.

It's not like I won't have room for another supply rail with the loss of the VGA connector, but it's added complication I suppose.

For normal DDR3, 1.5v, use this one -> MT41J128M16JT-125K <-, 128mx16 @ 2$.
This is a 1.6GHz part, can it be run at 300MHz?
256mx8 -> MT41J256M8

I wonder if there are any reduced latency DDR3 rams.  Cut down the RAS-CAS-DATA cycle.

Ah, that's in stock too.  Both good suggestions - I'd prefer the 8-bit one purely for easier routing and less IO headaches, but if the 16-bit one is going to provide better performance (i.e. allow me to use it for more MAGGIEs to access), then I'm sold.

[asmi]

Though you would need to run the device at 1.35v, or up to 1.45v.  Otherwise you would need 2 regulators to get the 1.5v and 1.35v for ram core.

No, you don't - all DDR3L devices are backwards compatible to "normal" DDR3 and will work just fine at 1.5 V.

This is a 1.6GHz part, can it be run at 300MHz?
<...>
I wonder if there are any reduced latency DDR3 rams.  Cut down the RAS-CAS-DATA cycle.

Yes you can, see attached table. It lists allowed CL depending on frequency. For 303 MHz (Tck = 3.3 ns), you can run it at 5-5-5 latency, CL=5, Trcd and Trp = 13.75ns / 3.3ns ~ 4.17 which rounds up to 5 (it needs to be in whole clock periods). Maximum frequency for CL5 is 333 MHz (Tck =  3 ns). For 400 MHz (Tck = 2.5 ns) you will need to go up to CL6.

[BrianHG]

Ah, that's in stock too.  Both good suggestions - I'd prefer the 8-bit one purely for easier routing and less IO headaches, but if the 16-bit one is going to provide better performance (i.e. allow me to use it for more MAGGIEs to access), then I'm sold.

The main maggies will always be 14 or 15.  The DDR ram will have 2-4 additional maggie layers (8bit vs 16bit) instead of the 1 which you would have gotten with the Hyperram.

At 720p, the main maggies would drop to 6 (maybe, -C6 could do it) and the DDR3 would drop to 1-2.

The DDR3 figures leaves ~half the bandwidth for the geo-blitter-audio-SD-Card HD DMA-Floating point ALU-Z80 access and I'm assuming full 800MHz controller.

(The truth is you can at least double/triple my DDR3 maggie figures, I've left room for massive overhead using a small burst size, but if you want more maggie layers for the DDR3 ram, for safety, you will have no choice but to introduce the ability for Z80 read wait states into your Z80 bridge code & use a larger burst size when filling the output display line buffer.)

[asmi]

Ah, that's in stock too.  Both good suggestions - I'd prefer the 8-bit one purely for easier routing and less IO headaches, but if the 16-bit one is going to provide better performance (i.e. allow me to use it for more MAGGIEs to access), then I'm sold.

Well x16 interface will provide you double the bandwidth at a cost of additional 12 IO balls. Whether it's worth to you or not, only you can tell. I would go for x16 as more bandwidth is always good, but it's up to you of course.

[nockieboy]

Okay, I'm going with the MT41J128M16JT DRAM.  Admittedly, for me it's almost a 'pin the tail on the parts list', as I don't really appreciate the nuances of the difference choices.  Just 'more RAM is good'.    Plus, if I have the IOs and can route a 16-bit DRAM up then that should be the goal, really.

Can you pls post latest schematics?

Almost missed this, asmi.  Latest schematic attached.