My children are both under the 5th percentiles for weight and height, and they have more or less been that way since about 4 months of age. DS is almost 5 years old and still weighs less than 40lbs. DD is 2.5 years and barely weighs 23lbs. They are both very happy, healthy and extremely energetic kids. Genetically they are just small. Were either you or your husband small as children?
One thing that helped me relax about DS' weight was to read through my baby book. When I realized his weight was tracking just a bit heavier than I did as a baby it helped reassure me. BTW I was formula fed as an infant, so clearly genetics not diet are at play.