Lets say room temp is 75 degrees. Your body temp is around 98 degrees. metal conducts heat very well so when you touch it all the heat is transferred to the metal whereas wood is more insulating and the heat from your finger leaves at a much slower rate. Also, there is also a difference in emissivity between the two materials. They radiate energy differently. The metal object not only feels colder in the room (or hotter in the sun), it really is a different temperature. Wiki/Google "emissivity."

Metal is a thermal conductor and Wood is a thermal insulator. When you touch the metal, the energy transfers rapidly to the metal, making it colder. When you touch the wood, the energy transfers very slowly from your hand to the wood.