Emacs 24.4 introduces the command delete-duplicate-lines .

By default M-x delete-duplicate-lines will delete all but one copy of any identical lines in the region. So assuming this is the region:

test dup dup one two one three one test five

it will be reduced to this:

test dup one two three five

When the command is invoked with a C-u prefix, it searches backwards and keeps the last instance of each repeated line. So this:

test dup dup one two one three one test five

will become:

dup two three one test five