You can also try clear plastic toilet plunger handles

that's right, toilet plungers. You can find these at dollar stores etc and they're usually pretty cheap. All you do is heat them to about 375 degrees,(on a non-stick cookie sheet) 20 minutes, remove from oven and stretch em out. They will taper down in the center and you will need to run cold water over them to lock in the shape. Then you break them in half (at the thinnest point) and presto! two of the most realistic icicles you've ever seen. There are little air bubbles trapped in the solid plastic just like real icicles.
You may have to sand the tips a bit and give them a coat of artificial water once you hang them on your scene but I'll guarantee they will look exactly like the real thing. I stumbled onto this a few years ago and my little granddaughters didn't dare touch one because they thought it would be too cold.
Oh yeah, I forgot to mention...a good pair of ove-gloves will come in handy, those suckers are h-o-t when you first take them out. This method is especially good for creating longer icicles, but you can make shorties too.
Good luck!