[Update: Initially I deleted this answer because I spotted it was already mentioned in the question. However I think it would be good for it to see votes so I have checked the Community Wiki box to prevent people from thinking I'm trying to game the system]

Apps from http://portableapps.com/ like Portable Firefox leave no trace on the system you run them on.

Also the PortableApps app itself makes using them a piece of cake.