<C-^> will switch to and from the previous edited file. On most keyboards <C-^> is CTRL-6.
3<C-^> will switch to buffer number 3. This is very quick, but only if you know the buffer number.
You can see the buffer numbers from :ls or from a plugin such as MiniBufExplorer.