An infamous $70 million unfinished mansion in Perth, nicknamed Taj Mahal on Swan, could finally be torn down, after sitting idle for years.

The Shire of Peppermint Grove has voted to demolish the now derelict building, which has been targeted by trespassers and graffiti artists since it was left half-built in 2011.

Fertiliser magnates Pankaj and Radhika Oswal spent more than $22 million snapping up eight lots to build their home, with plans for seven domes, multiple bedrooms and parking for more than a dozen cars.

However, after their Pilbara ammonia empire collapsed in 2010, the Oswals left Australia amid allegations of unpaid tax and construction on the site stopped.

Peppermint Grove Shire chief executive John Merrick said the mansion was now uninhabitable and had become a blight on the community.

"We know it's being frequented by itinerants and drug users, from time to time the police apprehend people there," he said.

"It's an unfinished building that's becoming the target of graffiti and it's most untidy."

The Oswals have 21 days to appeal the demolition order to the State Administrative Tribunal.

A statement from the Oswals' lawyers said Mrs Oswal filed submissions on January 19 in response to council concerns.

The statement said: "Mrs Oswal is disappointed that the Council has not informed her first of any resolution passed prior to contacting the media".

It also said Mrs Oswal would continue to engage with contractors to deal with health and safety concerns raised by the council.

Mr Merrick said the council would fight for a resolution to the dispute, one way or another.

"If the appeal is successful it's just the start of many things because of the fact the building licence has expired, the property has become derelict, and it's uninhabitable as determined by health and building surveyors," he said.

"So we will continue to pursue the completion of the building, or the demolition of the building.

"It's become a blight on the landscape, and it's dangerous to trespassers who decide they're going to break the law and visit the property."