Sydneysiders paying their respects to siege victims at Martin Place have shouted down a group of anti-Islamic protesters who appeared at the memorial on Friday afternoon.

Four men waving Australian and Southern Cross flags shouted slogans including "Islam is evil" and "not all cultures are equal - Islam is inferior".

The men also accused Muslims of being "murderers" and supporting female genital mutilation, according to a witness.