Are you keeping a daily log? Normally I wouldn't suggest watching the clock, but perhaps taking note of when he nurses, and for how long. This may give you a better idea of his eating habits (also it will give you something tangible to show the doctor). Perhaps he's only eating for a few minutes, but has 14 feedings a day, kwim?
Have you tried offering the breast when he isn't showing signs of hunger? This may encourage him to eat more
About the drugs/alcohol question, yes, it's possible for drugs/alcohol ingested during pregnancy to have a continued effect on a child. However, has the baby been diagnosed with FAS or anything related to that?
It may just be that she is tiny, and will always have a smaller stature Do you know the "size" of her parents? (tiny-framed adults produce tiny framed babies )
ETA - the most imporant thing is that she is "thriving". You want her to be healthy and happy.