Why does quantum mechanics need an infinite dimensional vector space?

When undergraduate physics students are trying to learn quantum mechanics, one of the most difficult obstacles is that abstract infinite dimensional vector spaces are needed, which makes it impossible for a person to mentally “see” the geometry in their minds in the way a three-dimensional space can be imagined. The fact that the coordinates of the points in that vector space are complex numbers certainly doesn’t make it any easier. When I was studying that myself in the early undergraduate studies, it took me over a year to get a hold of the concept that any self-adjoint operator acting on vectors in the quantum state space has a set of eigenvectors that form a basis of that space.

Sometimes, however, a quantum mechanics problem can be handled by approximating the state space with a finite-dimensional vector space. In fact, several physics Nobel prizes have been awarded for the study of two-state quantum systems. One example of a situation, where a quantum system is effectively a two-state system, is something where the ground state (n=0) and the first excited state (n=1) have energies that are close to each other, and the energy gap between the first and second (n = 2) excited states is unusually wide. Then, if the system is at low temperature, the Boltzmann factors of the lowest-lying two states,




are much larger numbers that the Boltzmann factors of any other states, and it is unlikely that a measurement of total energy will give a result other than one of the two lowest energy eigenvalues.

How, then, does one know that a quantum state space actually has to be infinite-dimensional? I once saw, in a mathematical physics textbook, a practice problem where this had to be proved, and I was proud for almost immediately finding the correct answer.

A cornerstone of nonrelativistic quantum mechanics is the position-momentum commutation relation


Now, if the quantum state space were finite-dimensional with dimension N, then these operators \hat{x} and \hat{p} would be N\times N square matrices, and the constant i\hbar on the RHS would be a diagonal matrix where all the diagonal elements have value i\hbar

Actually, this equation can’t hold in the finite-dimensional case. To see this, consider the calculation rules for taking the trace of a matrix sum or a matrix product:



Now, if you take the trace of both sides of the commutation relation equation, and use these rules, you will get the result


Which clearly isn’t possible. So, we have done a false assumption somewhere in the derivation of the contradictory equation above, and the false assumption must be the finite-dimensionality of the space where x and p are defined. An infinite dimensional operator doesn’t always have a trace at all, and this is the case with the operator xp-px.

An interested reader may try to form finite-dimensional matrix representations for the raising and lowering operators of a quantum harmonic oscillator, use those those to form matrices for operators \hat{x} and \hat{p}, and see what happens when you calculate their commutator.