I'm no doc (god help you all if I were), but I seem to recall that ginko biloba is a mild stimulant and has some un-verified chemical properties. Consequently, it was frowned upon--if not outright banned--by the docs. I do remember being told by a doc not to take it though. Someone (feddoc I think) posted a good NAMI link that outlined banned substances a while ago, so I suggest you do a quick search for that thread.
And I've never heard of ginko biloba being used to enhance color vision or depth perception before. The girlfriend (admittedly NOT a flight doc, but a pharmacist) says it's mainly used as a memory-enhancing substance, but that there's no direct evidence, FDA or otherwise, that suggests the stuff actually works, though. I forgot to ask about ginseng.
Look for that thread and the corresponding link, though.