Okay, so the key kernel source file is
v2.6.26.8/drivers/video/aty/radeonfb.h,
v2.6.26.8/drivers/video/aty/atyfb_base.c, and
v2.6.26.8/drivers/video/aty/radeon_base.c.
The comment in
radeon_base.c:radeonfb_check_var() indicates the driver only supports
8-bit grayscale,
ARGB555 (ignoring \$a_0\$),
RGB565, and
ARGB888 (ignoring \$a_0\$ through \$a_7\$). Because these values are what you see in
struct fb_var_screeninfo but do not correspond to the actual video mode the driver sets the GPU to, it is the mode-set code that is faulty. Which is not that surprising, because DRM-based radeon support was mainlined by
kernel 2.6.31, and most users and developers moving to that instead.
Again, note that the Linux kernel drivers do not expose all the actual hardware functionality, only what the developers have found necessary/viable/useful.
However, the v2.6.26 driver definitely supports both Standard and Doubled green gamma approaches for supporting 15-bit RGB via RGB565/555. The
FBIOPUTCMAP ioctl invokes
v2.6.26/drivers/video/fbmem.c:fb_ioctl(), which ends up calling
v2.6.26/drivers/video/fbcmap.c:fb_set_user_cmap(), which ends up calling
v2.6.26/drivers/video/fbcmap.c:fb_set_cmap(). This one uses the framebuffer-specific
(struct fb_ops)->fb_setcmap() if it exists, and
(struct fb_ops)->rb_setcolreg() for each entry otherwise. On ATi Rage128, this is
v2.6.26/drivers/video/aty/aty128fb.c:aty128fb_setcolreg(), on Radeons it is
v2.6.26/drivers/video/aty/radeon_base.c:radeonfb_setcmap(), and on other ATi GPUs, it is
v2.6.26/source/drivers/video/aty/atyfb_base.c:atyfb_setcolreg() (where the comment is incorrect: supplied
red,
green, and
blue values are in 0..65535, not scaled to hardware at all).
In all cases (even for DRM emulated framebuffers), for the RGB565/555 and BGR565/555 modes, you use
FBIOPUTCMAP with a
struct fb_cmap that has
.start = 0,
.len = 64,
.transp = NULL, the 32 initial entries in
.red and
.blue containing the gamma ramp from
0x0000 to
0xFFFF, inclusive, same repeated for the latter 32 entries, and all 64 entries in
.green defining the green ramp, also mapped to
0x0000..
0xFFFF. For standard green gamma approach, the ramp value for each odd entry is the same as each preceding (even) entry, so that the least significant green bit is ignored, and this is the most widely compatible approach; for doubled green gamma approach, the ramp values for
.green are the same they are for
.red and
.blue, so that the most significant green bit is ignored.
In practice, for your non-DRM radeonfb framebuffer, it could be possible that the gamma ramps are just incorrect, programmed to the doubled green gamma, and doing the
FBIOPUTCMAP with proper gamma ramps fixes it –– whichever format (RGB565 or RGB565/555) you do want to use.
(In general, you want a gamma curve where the first nonzero empty differs from all-black, and halfway has the same perceptible brightness than a checkerboard pattern of full-intensity and black pixels. Usually the ramp value at index \$0 \le i \lt N\$ is calculated using \$65535 \frac{e^{\gamma i/(N-1)}-1}{e^1 - 1} \approx 38139.84 \left(e^{\gamma i/(N-1)} - 1\right)\$, but due to peculiarities in human perception, the first few darkest values are often linear instead. The difficulty in using such calibration images is that you have three adjustable values for red, three for green, and three for blue: first value sets the gamma (\$\gamma\$, default is 1.0), second the number of linear darkest entries, and the third the intensity of the second-darkest (darkest away from black) entry. Most humans are not analytical enough to get "there" by themselves, adjusting these nine values, because there are often some tradeoffs, and the optimum set does depend on the ambient brightness and display brightness too.)
The same applies, if you only need 12 bit color, but have an use for the four extra bits per pixel, but do not want them to affect the visible color. Then, you program RGB565 (all devices that I've seen supporting BGR565 have also supported RGB565), and set a 64-entry
struct fb_cmap where each color ramp repeats four times. Then, your 16-bit pixel colors are
0bWrrrrXYggggZbbbb in binary, where
W,
X,
Y, and
Z are the unused bits. They are not contiguous and cannot be contiguous using standard Linux framebuffer userspace interface, unfortunately, unlike in
ARGB444 or
RGBA4444 modes.
(ARGB444 is particularly nice in that if you use \$a_0\$ through \$a_3\$ as a 4-bit depth/Z, you can do very fast Z-buffering by simply reading the old 16-bit pixel value, and writing the larger of old and new unsigned 16-bit pixel values, essentially
*pixel=max(*pixel,value).)