Facebook Inc. and Twitter Inc. are building up their forces to fight an emerging enemy: "social" spam.

One of their foot soldiers is Tao Stein, a Facebook engineer. At 4 a.m. one May morning, Mr. Stein was jolted out of bed by a spam alert on his cellphone. Facebook was being inundated with messages that read, "hey check out this link FREE IPAD." But there was no free iPad—just malware that caused Facebook users who clicked on the link to unintentionally rebroadcast the annoying message to friends.

Mr....