Five 12 volt downlights - can I take 3 out of the circuit and just use 2 with the transformer?

In short yes.

If you took three of the bulbs out the remaining two would still come on, so on that basis it is perfectly ok to remove three lights completely and run the remaining two from the same transformer. The switch wire should be connected to the circuit in front of the transformer so you don't have to worry about disconnecting the light that would normally have the switch wire connected to it (if they were 230v lights).