Exactly. Each voice card had its own 6500 (or 68000 for the Series III) processor and its own RAM (128k per card on the first rev, I think), which is what made them so expensive. The output DAC clocks were able to go up to 192kHz, and could be very precisely controlled. They couldn't figure out how to make a common RAM pool for all the voices, though, which is why each card was basically self-contained, and also why the system was so expensive. There was also a central control computer, which ran the interface and controlled the cards.
The big sampler breakthrough after that was from Emu, who had figured out how to have each output voice read data from a common RAM pool. All of a sudden, you needed one processor instead of nine, and a lot less RAM. This is why the first Emulator cost $8,000 instead of $30,000. They also opted to use a resampling engine to change the pitch of samples on the fly before output, which didn't sound as good as the Fairlight. Of course, these days that's a moot point, with computing power high enough to run really good resamplers.
The real beauty of the Fairlight, according to people who used it, was really in the software and interface, rather than in the hardware itself. It had software on board where users could sculpt sound in a variety of ways, from drawing waveforms to drawing envelopes for harmonics and so on, and the computer would then calculate the samples to play back. Pretty slick stuff for the time.