Should you call the police?

No. Unless there is dealing going on in your neighborhood or other criminal activity that is directly affecting you, just let it be. Smoking marijuana may be illegal under most conditions in the U.S., but it really isn't any worse than drinking alcohol or smoking cigarettes.

How would you feel if someone followed you around and got you arrested/ticketed/sued for every minor legal infraction that you committed, whether it personally affected them or not? If you want to make enemies quick, sicking the cops on your neighbors is one way to do it.

Will they be able to tell without going into the neighbor's house?

Maybe. But if you have neighbors that burn leaves, it's possible that they're unintentionally burning ditch weed (wild marijuana) in the process - in which case the smell could still be coming from them.

Would you be able to smell it coming from them at all?

Maybe. But my guess is that if you don't actually know what marijuana smells like, then you're a bad judge of whether this is marijuana and whether or not it is coming from your neighbors. "Acidic" and "unpleasant" are not common descriptors applied to marijuana smoke.

Response to additional details: The idea that marijuana is "frequently only the beginning" is based on nothing other than rumor and fearmongering. There are no good studies indicating that marijuana is truly a gateway drug. I know plenty of people who have never had a cigarette or a drink of alcohol - much less anything harder, but smoke marijuana everyday.

Some people have legitimate medical reasons to smoke marijuana, and depending on where you live, that could include your suspected neighbors.

If you like your neighbors (naughty or nice), then you ought to feel comfortable enough with them to ask if they're smoking marijuana, and if so, then could they please close their windows or use fans or brainstorm some way in which the smoke would not affect you and/or your baby.

By the way, you and/or your baby would not get stoned from secondhand smoke. THC dissipates quickly, and certainly more quickly than it could travel between houses. You might need to be concerned about affects of second hand smoke - but in that case, it would be the same if your neighbors are smoking cigarettes.

Regardless, just go talk to them! Calling the cops in this situation, given that you know and like your neighbors, would be like my calling the cops to tell on my roommate for her playing her stereo too loud, when I could have just walked across the hall and asked her to please turn down the volume.