We took the luxury bus from The Empress hotel - the driver was great and you get to learn a little about Victoria and the price includes the gardens admission. It saves you money to do it this way if you don't have a car. The gardens are beautiful and the food was great in the self serve dining room.
