There are two places i ALWAYS find harvesting points. both are in W. Saru. and both are over between Giddeus and the Outpost. if you take the path from the outpost to Giddeus, there are two raised-ish areas. one is just off to the left of the outpost (it's visible from the outpost itself), and the other is along the right side of the path, heading towards the outermost horutoto tower. Now, i don't frequently get cotton anymore (i always seem to get Wijnruits and Flax Flowers >_<) but i usually manage to head back to Windurst with 4 or 5.
As for Giddeus... well, i've never once found a harvesting point there, and i've been all over the place. friends of mine swear that there are some there, and i believe them, i've just never managed to find one.
Honestly, if you need the cotton, stick to W. Saru and bring lots of sickles. if you need the money, go excavating in Tahrongi (turtle shells and chicken bones are far more lucrative).
~Hikaze of Diabolos.