In 1844 Marx wrote "Rent of Land", talking about how landlords are more or less vampires on society, taking a percentage of all value created on their land simply for owning it.

However, as you can see if you read it, it is more or less just quoting Adam Smith and agreeing with him, with minor squabbles. One of the large motivations for Adam Smith writing The Wealth of Nations was to convince people to try to move capital away from the unproductive landlords (who at that time were mostly comprised of very wealthy landed English gentry) into the hands of entrepreneurs and workers. He noticed two things about landlords, first, as he puts it "The landlords, like all other men, love to reap where they never sowed, and demand a rent even for the natural produce of the earth."

That is to say, they demand a percentage of what is extracted from their land, despite having done no work themselves to create that value. Adam Smith was more concerned with landlords that extracted a percentage of money from businesses than individuals renting houses, which at the time added up to a much larger amount of money being drained away (probably still true, but less so). He gives an example of the sudden discovery that a certain type of seaweed was useful in the production of alkaline salt. Landlords whose land happened to neighbor the sea where this type of seaweed grew immediately increased their rent, despite having done nothing to earn such money. They didn't even own where the seaweed was extracted, being in the ocean, but the adjacent land was needed to extract it, so they could blackmail entrepreneurs who used the seaweed for rent extraction.

The second thing he noticed about landlords is what they do with the money they extract. Unlikely entrepreneurs, they don't reinvest it into productive activity. A farmer, with their profits, tends to use a good portion of the money to improve their farm. They buy newer, more efficient machines. They hire help. They purchase seeds, they try to improve the soil, etc. All of this, over time, amounts for greater total production and a wealthier nation. The landlord who owns the land that the farmer rents from however, has nothing to improve. He makes money from taking money from the farmer, because he owns something. So what they tend to do was spend the money on themselves. Landed gentry in England would spend their money throwing fancy balls and purchasing luxuries. Entrepreneurs would spending their money to improve production. Therefore, he argued, if more of the money was diverted away from landlords, production would grow at a faster rate. In fact, Adam Smith noticed that the main way land increases in value is for other people to build things around it. For example, if a train is built next to a plot of land, suddenly the landlord will find his land more valuable, despite no effort of his own, and increase rent.

In this way Adam Smith thought that at least, while landlords produce nothing and contribute nothing and are generally a drain on society, their interests are at least aligned with society because they have a general interest for society to improve, since any improvement around them improves the value of their own land. This is where Marx disagrees, because despite this fact landlords will always have an interest against their tenants, in that they want to take as much value as possible from their tenant, so their interest in general run opposite society. This can be seen today in that landlords have a general interest for housing prices to rise continuously, whereas it would be better for society in general if housing prices fell continuously, so people could afford homes. This is why landlords will be extremely politically active to fight against any measure to make housing affordable, because it will decrease the value of homes and ultimately rent.

In short, when someone tells you that you are a dirty communist for saying landlords contribute nothing and merely drain society, remind them that the idea came first from Adam Smith.