The Australian Christian Lobby (ACL) says the prime minister's decision to pull out of a speaking engagement to the group's supporters will be seen as an "abandonment of the Christian constituency".

Prime Minister Julia Gillard on Thursday withdrew from speaking at the ACL conference in Canberra on October 5 and 6, citing "offensive" comments about homosexuality made by its leader Jim Wallace, who compared smoking to same-sex marriage.

Ms Gillard, an atheist, was due to address the conference on religious freedom in a secular democracy and said the organisation was entitled to its views.

"To compare the health effects of smoking cigarettes with the many struggles gay and lesbian Australians endure in contemporary society is heartless and wrong," Ms Gillard said in a statement on Thursday.

"In light of this, I believe my attendance at the conference would be inappropriate."