I'd say...
First, read a book on basic electronics. You want some idea of what the fundamental components do (resistors, capacitors, inductors, diodes, transistors) and the basic laws.
Second, get the basic tools: soldering iron, multimeter, breadboard & jumper wires. The multimeter does not need to be expensive, even the very cheapest ones work pretty well usually.
After that it gets less clear....
I like to use multicolored wire when I build stuff, as it makes tracing problems through a rat's nest of wires easier. And you're going to need wire. Places sell assortments of different-colored hookup wire, in solid and stranded varieties. The solid wire will work in breadboard holes--not REAL good, but it works.
Beyond that it really depends on what areas you are interested in. Some areas of electronics use parts that you hardly ever seen used in others. And even then, there's no point in buying every part you see. You don't need vacuum tubes unless you are building a tube amp or a vintage-style HAM radio--and even then, there's a few good tube choices for those uses, and lots of tubes that aren't really very good even if you can get them for free.
When starting out, I don't think it's excessive to buy general assorted packs of basic parts like resistors, capacitors and inductors, but keep in mind that electronics hobbyists tend to be habitual pack-rats. (-And I can't say I'm totally innocent of that offense...

) Every OLD electronics geek tend to have a HUGE pile of parts that ends up getting sold at their estate sale. There is always a temptation to buy more (because the cost-per-part is lower) but if you want to minimize the clutter, it may be better to just buy what you need as you decide on particular projects.