For a real helicopter the main rotor frequency (1R) is of the order of a few Hz. The blade-pass frequency (nR) will be two, three, four, or five times higher, depending on the number of blades. The components of interest for RTB are the vertical and lateral vibration at each frequency, generally expressed as a vibration velocity, even if measured as acceleration. A tri- or bi-axial accelerometer mounted on the airframe somewhere near the gearbox mounting is used, often in a location specified by the aircraft manufacturer. Obviously this picks up a lot of other vibration sources, from the tail rotor/fan, tail drive shaft, engine shaft, gearbox, etc. There's a lot of vibration about.
To get rid of this stray vibration, synchronous averaging is used. A pickup on the main rotor shaft gives an impulse, for example each time the master blade is over the nose. This is used to synchronise the data acquisition system so that it samples the accelerometer signals at equal increments of rotor angle, rather than equal time periods. Obviously in a full size helicopter the rotor speed is nearly constant: but not exactly. You only need a few samples for each period of the blade-pass frequency. Sixty samples per revolution is a good choice, because 60 divides by 2, 3, 4, 5, & 6 exactly. Back in the old days we generated a variable-frequency ADC sample clock using a PLL off the main rotor tacho, but then moved on to constant-frequency ADC operation and variable-ratio resampling.
The corresponding resampled sample points from successive rotor revolutions are averaged together, over maybe 30 seconds or so, while the pilot holds a steady flight condition. The averaging acts as a bandwidth-narrowing operation, filtering all signals which are not rotor harmonics, and accumulating those which are. This measurement & averaging is applied to each of of the two or three axis signals simultaneously. Then you take the FFT of each averaged signal (not the power spectrum) and extract the amplitude and phase of the 1R and nR bins. So each flight condition gives you 4 measured numbers, per axis. Note that the FFT phase is the phase relative to the rotor shaft tacho impulse - effectively the rotor shaft angle.
You repeat this process over a set of flight conditions - FPOG, hover, and various forward speeds. Sometimes extra conditions such as autorotation or HOGE are used as well. You then have to make adjustments to the rotor system to reduce the amplitude of 1R & nR in each axis at each test condition (generally you don't care about the phase). The available adjustments are the adjustable pitch links in the control system; one or more weight pockets at each blade tip; sometimes one or more tabs on the trailing edge of each blade; and possibly also lag dampers (though these are generally a case of replace rather than adjust).
At the same time, in each flight condition, you measure the tip-path of each rotor blade. In theory, they should all follow each other exactly moving in the same plane, but in practice some blades fly high and some low. Similarly, some blades may lag more than others (i.e. arrive slightly early or late at the reference position compared to the ideal). This measurement is generally done optically. This gives you another set of numbers at each flight condition, all of which need to be minimised. I spent years working on optical blade tracking devices!
There are more things to be minimised than available adjustments, i.e. the system is under-determined, meaning that a perfect solution may not exist. We used to treat it as a constrained linear least-squares system, since in many cases there are absolute limits to be observed. There may be alternatives techniques these days.
The first RTB system I was involved with used a Rockwell 6511 to control the data acquisition and a National NSC800 to crunch the numbers (both 8-bit microprocessors!). This was possible because you have to land & shut down before the ground crew can make the rotor adjustments, so there is always a delay of a few minutes. Then we moved on to a pair of 68000 16-bit micros. That system went out to the first Gulf War almost before we had got it into production. Then we moved on to HUMS systems, where the RTB function was secondary to monitoring the gearbox & engines.The first one of these used Inmos Transputers (yes, really!) after which I left the company and moved on to other things.
I remember shortly after the collapse of the Soviet Union, flying in Switzerland in a Kamov coaxial-rotor helicopter while we were measuring the effect of introducing specific rotor system adjustments on the measured vibration - effectively calibrating our least-squares solution. We were accompanied on one flight by a senior Kamov engineer, who was very old school. He didn't like the level of vibration he was feeling in one of our tests, and decide to check it out. He produced a piece of card and a pencil. Then he leant against the airframe with the arm holding the pencil, and moved the card down past it with the other hand, drawing what could have been, but was not, a straight line. He then measured the peak-to-peak amplitude of the wiggles, and used his slide rule to convert from displacement to velocity. I don't speak Russian, so I don't know what his measured value was - a pity, as it would have made an interesting comparison.
----
For an active vibration control system such as the one you are proposing, you will need to generate a force which is ideally equal and opposite to the force from the rotor imbalance at each flight condition, and apply it at the same point, i.e. the gearbox output shaft. The force must have components at the 1R and nR frequencies with independently adjustable amplitude and phase, controllable separately in the vertical and lateral axes. Obviously you can't do this with a single motor. However, you can in principle at least, cancel out one component - I suggest the 1R vertical. You will need to measure the phase angle of the rotor both so that you can isolate & measure the 1R vertical, and so that you can control the motor speed so that it rotates synchronously with the main rotor - effectively a PLL where the 'voltage controlled oscillator' is your DC motor variable speed control. Once this loop is in lock your motor should behave as though it was coupled directly to the rotor shaft. Then you need (somehow) to make this motor generate a variable periodic force as it rotates - effectively you want an adjustable imbalance in the motor shaft. Your control system must adjust both the imbalance and the phase of the motor shaft relative to the main rotor shaft so as to cancel out the chosen component. Tricky!
One way to make a varying imbalance might be to use a flexible shaft with a weight attached to it, between a the motor an a moveable bearing. By moving the bearing, you could introduce a misalignment in the shaft so that the weight would generate an out-of-balane force that would be (nearly) zero when the bearing was aligned with the motor shaft, and increases as the angle between the two bearing increases. Maybe - I haven't worked this out in detail.
Good luck!