You could use a Joule Thief or flyback style blocking oscillator. To boost the 1.2v from the panel up to 5v or more (easy to get a few hundred volts from 1.2v@20ma) in a capacitor. Then use a transistor and LED to discharge the cap in pulses to the battery. The Led gives a voltage differential before the transistor turns on, then there is a path of lower resistance, and the cap drains to battery voltage. at which point the transistor switches off, and the cap fills again. I can do thin in frequency from a few hundred hertz, to hundreds of kilohertz. Depending on rate of power supply. Ie it goes slower on less input power. Or its possible to charge, just putting the inductor pulse output directly across the battery too. Still testing to see which way is most efficient.
Its not perfect. and still in experimental stages. I have been playing with this system with many batterys for a few months now. I can take a little more amps, at a lower voltage and give a 12v lead acid a maintaining trickle charge this way. and am getting good results from AA's C's D'c etc As well as my home made lead alum batterys. Most batterys seem to respond well to this type of charge. Problem is your Phone and battery have inbuilt charge controllers that wont let it work. You can get DC 5v no worries. but the amps is pathetic. Probably want do much charging, if it does it will be really slow. I have seen other people charge LI-po and Li-ion this way, but they normally take the battery out of the device, and cut it open to remove the controlling circuitry. Not advisable if you dont know exactly what you are doing. Li-po's can burst into flame or even explode. If over charged.
Up side is, if you get it right, you can charge and recover batterys that have done many cycles and no longer charge. I have recovered Ni-cd's and Ni-mh this way. So they charge on comercial chargers that previously rejected them. I have seen similar things done with lead acid and various lithium cells. Its tricky getting a decent efficiency though.
It is do-able, but its a bit advanced. Not for beginners.
I'm working on a device that will charge Phones and MP3's etc, via USB from flat disposables, or ambient (or direct) sunlight. I will likely be incorporating some batterys in the device though. To store up the "harvested" power, for solid dc output to a device to be charged, without modifying the device..
There are 5v panels on ebay that will do what you want. Some even have internal batterys, so you can save some charge for after dark.