How does luminance work?

[Connecteur]

When I adjust contrast, it seems to change brightness, and visa-versa.
What am I missing?

[Zeyneb]

Are you talking about a computer monitor?
I think it is useful to elaborate slightly more on what you have and what you're trying to achieve.

[DL2XY]

Contrast is gain, brightness is offset.

[Connecteur]

Any video display with both brightness and contrast controls.

[Nominal Animal]

For simplicity, consider a grayscale image, with each point on the display having grayscale value L between 0.0 (black) and 1.0 (white).  (I will also simplify actual details and operation here, so you can get a correct intuitive picture; so consider this a very rough approximation, and not an exact description of reality.)

Let's say we have contrast C between 0 (image is all grey) and +infinite, with 1.0 being "normal".  Infinite contrast is when each point on the display is either all black or all white, with nothing in between.  (So, in reality, the actual range might be limited to say 10.0 or so.)

Let's say we have brightness B between -1.0 (it's all black) and +1.0 (it's all white), with 0.0 being "normal".

The actual displayed grayscale values L' are essentially modified via something like
        L' = (L + B)^C - B
except that L' is clamped to the displayable range, say 0.0 to 1.0.  (For completeness, we define 0⁰ = 1, so that x⁰ = 1 for all real x, including zero.)

How does this show up in practice?

Adding contrast C makes dark areas darker, and bright areas brighter.  It does not affect midtones.  Reducing contrast decreases the difference between dark and bright areas, so that with zero contrast, the entire display is uniform gray.

Brightness B adjusts the brightness of all tones.  Increasing brightness makes dark areas, midtones, and bright areas all brighter.  Decreasing brightness makes them darker.

If you have zero contrast, brightness changes the brightness of the uniform gray you see.  If you have infinite contrast, brightness does not affect the image at all (because dark areas are as dark as the display can display, and bright areas are as bright as the display can display.)

Digital displays, like LCDs, plasmas, LED, OLED etc. do this slightly differently.  LCD (like TN and IPS panels on laptops and many TVs) is transmissive, and the maximum brightness is dictated by the brightness of the backlight.  Some devices might be able to vary that brightness, others cannot, so instead of the formula shown above, they use a different approximation that produces similar visual results.  With LED and OLED technology, each image pixel produces light, and with digital data, the entire adjustment is digital – either the formula shown above, or something similar (possibly look-up tables implemented in the HDCP transmitter, or DP/DVI/HDMI processor).

Analog displays, especially CRT displays (old-style monitors and TVs) adjusted point brightness by varying the voltage of the electron gun.  There, contrast adjusted the total range of the voltages applied, and brightness the bias (or center/midway voltage).

What about color?

See e.g. the Wikipedia article on photoreceptor cells.  Essentially, current displays reproduce color using three additive color components: red, green, and blue.  (Either the transmissive filtering layers pass through, or the light sources produce, light of one of these three colors for the human eye to perceive.) This is called the RGB color model.

In analog displays, the electron beam excites a phospor coating, and the simplest possible brightness and contrast affected all three color components.

In digital displays, each of the three color components could be treated separately as L above.  In practice, however, they tend to have their image buffers use a different color format, typically YCbCr, where the Y component is essentially luminance, and matches L above, with Cb and Cr corresponding to the color.  On input and output, conversion between RGB and YCbCr is a simple mathematical operation.  It is usually written in matrix form, but don't let it fool you:
$$\left[\begin{matrix} R \\ G \\ B \end{matrix}\right] = \left[\begin{matrix} m_{11} & m_{12} & m_{13} \\ m_{21} & m_{22} & m_{23} \\ m_{31} & m_{32} & m_{33} \end{matrix}\right] \left[\begin{matrix} Y \\ Cb \\ Cr \end{matrix}\right]$$
is just shorthand for the three simple equations,
$$\begin{aligned}
R &= m_{11} Y + m_{12} Cb + m_{13} Cr \\
G &= m_{21} Y + m_{22} Cb + m_{23} Cr \\
B &= m_{31} Y + m_{32} Cb + m_{33} Cr \\
\end{aligned}$$
The matrix form is useful, because inverting the matrix (using inverse or pseudoinverse as described in Linear Algebra) gets you the inverse operation.

[Bud]

Contrast control changes signal amplitude swing. Brightness control does not change that but instead it changes how far from the ground level that signal is sitting. Ground level being equivalent of Black.