A lot depends on what you're going to be doing.
If you were interested in digital electronics, I might recommend buying some 1k, 10k, 100k, and 1M resistors because when you use resistors their exact value is often not important.
My suggestion is to check to see if they have packs of assorted resistors and similar for capacitors and buy them.
I think I bought a pack of 500 assorted resistors about 20 years ago (it may have been 30 years ago) and they lasted me (for breadboarding) up until a couple of years ago when I bought another similar pack. When my eyes were better I could pick out the right resistor fairly easily, these days it's not quite so much fun...
Capacitor packs tend to come in at least 2 types. One will include larger value electrolytics, the other smaller values. The packs I bought years ago still have remnants in my breadboarding "bin".
The breadboarding bin gets refreshed from time to time as I have the odd component left over from something, and in it goes.
If I were to be really organised, I'd probably split up the resistors by decade (1-0.0, 10-99, 100-999, 1000-9999...) just to make it easier to find values faster.
Some people have suggested doing this split by E12 value, so you have in one bag 1, 10, 100, 1000, ..., and in another 1.2, 12, 120, 1200, ... and so on. The problem with this is that it's fine if you have E12 resistors (or only E12 values) and all you need to do is look for the correct colour multiplier, but t breaks down when you get E48 or E96 values which mean *way* more bags. The other problem is that when you go from 2 stripes plus multiplier to 3 stripes plus multiplier, the colour of the multiplier changes for the same value resistor (10k = 10 * 10^3 -- 3 bands, or 100 * 10^2 -- 4 bands).
Jaycar have 4 packs of resistors. I would probably start with the 0.25W carbon film pack with 300 values for $6.95 (RR1680). I started off with something like their 1% metal film pack which is $14.95 for 300 resistors (RR0680). It's unlikely that you'll need the additional accuracy for breadboarding things, and if you need the additional power handling you can place a couple of resistors in series or parallel -- and that won't happen often. (The packs are on Page 73 of their catalog which costs $3.95 or you can get one with this month's Silicon Chip magazine.
They also (Page 77) have 4 packs of capacitors. the electrolytic pack RE6250 ($11.95) contains 55 capacitors from 1 to 470uF (they will be various voltage ratings), and the greencap pack RG5199 (which are 100V and not many are green any more!) are $9.50 for 50 values between 0.001uF and 0.22uF.
Somewhere in the past I also bought a pack of transistors. On Page 65 they have ZT-2170 $14.95 which has 100 common small signal transistors. For most uses you can split them into PNP and NPN and just pick one at random.
Stuff for breadboarding is rarely critical (in fact very few components are critical in any design) so you can save some money by getting some of this stuff online. For low voltage use, buying resistors, capacitors, and transistors in packs like these from China is not likely to cause huge problems. Breadboards themselves can be very cheap to get from China, and although they're not the best quality, it can sometimes be useful to have a couple of them if you don't want to pull one circuit apart to make another.
From ebay
--
1000 1/4W 1% resistors for $10 (search for "assorted resistors")
--
500 electrolytics for $16 (search for "assorted capactors")
--
100 transistors for $5.00 (search for "assorted transistors")
--
Solderless breadboards 830 point for $6 (search for "solderless breadboard")
--
Jumper wires 65 for $3.30 (from search above)
The jumper wires are especially useful. They save you from having to scrounge up pieces of wire. If you want to scrounge them up, 1m of UTP cable (8 solid cores) will last you for ever. Just make sure you don't get the flexible UTP cable.
edit: if you go the chinese ebay transistors, pick a selection of the 2Nxxxx transistors as I'm not sure exactly what the other ones are, and it's important to know what you've got and to be able to find specs on them.