Fonts have always been a curse on X11.
No, not really; there's just been quite a few changes along the way, making things muddled if you aren't aware of the history.
Because of the client-server separation, there are two sets of fonts: core, and client side. Most applications nowadays use client side fonts. If you use a thin client, you'll need to disable font antialiasing, so that text is rendered on the server (machine containing the display hardware; what we normally consider the client) using the Render extension, with the minimum amount of data transferred. With antialiasing enabled, the pixel data will flow back and forth between the server and the client.
You definitely want to ensure you have Render extension enabled (and EXA for Matrox Xorg drivers), and are using Xft2 and Freetype.
Twenty five minutes to load 11Mbyte of fonts!
Make sure you have just the bare minimum core fonts (
/usr/share/fonts/X11/), and prefer Truetype client-side fonts (
/usr/share/fonts/truetype/).
Old applications (on top of bare Xlib) may use the core fonts still, but Gtk+ and Qt (so Gnome, Cinnamon, LDXE, XFCE, and KDE) applications will use client-side fonts. If you disable font antialiasing, the Render extension will be used to render them.