Hi, thanks for all the replies.
I made a smaller/simpler circuit just with the arduino board, power supply, three leds and the phototransistor.
Since I can't hook it up to my PC. I made a small sketch which measures the sensor and calculates the difference with the previous reading. If the delta is smaller than 2, no led will light up, between 2 and 5, one led will light up, 5 and 10 the second will light up, larger than 10 the last one.
It's quite crude but since I don't have an LCD display to hook up it's the best I can do
1st test: with 9v battery connected to RAW: as expected , very stable from time to time the first led lights up
2nd test: 7.5V wall adapter connected to RAW, leds/sensor being fed by 7.5V: there it is, erratic readings on the ADC, leds flashing randomly
3rd test: 7.5V wall adapter connected to RAW, leds/sensor being fed by VCC (so from arduino power regulator) : much better, slightly worse than 1st test
So I guess I have to improve the LM317 circuitry (larger caps?) and also hook up the arduino on it's own power regulator as suggested by Arne.
So it's clear the problem is related to the power supply. I attached a scope, there's a sawtooth ripple 20 mV peak to peak, on the rising slope I can see 4 spikes that measure 60mV. The frequency of the ripple is around 300Hz.
As a last test I repeated the 2nd test, but this time I hooked up ONE of the free GND pins to mains ground, meaning that the power supply/circuitry is no longer floating. The readings improve significantly, a little bit worse than test number 3. This is what I fail to understand, why does it improve? On the scope I see the same ripple, the peak to peak voltage is slightly smaller and the spikes are around 40mV (also slightly better)