Mildew grows wherever there is high humidity. It doesn't grow on the vinyl fence itself, but on the dirt and dust which sticks to the fence and gives the mildew a medium to grow. You can get rid of it by spraying with a diluted (50/50) bleach solution. Or, by using a pressure washer, or simply scrubbing with soap and water using a sponge or brush.