It's possible that it was Mjolnir itself that dented the shield. It's a very impressive weapon, and as Gorton says: It's an unstoppable force.

Odin, chief of the Norse gods, ordered the dwarfs to forge Mjolnir. The blacksmiths Eitri, Brok, and Buri used the core of a star as the mold. In the process the star exploded and almost took Midgard (Earth) with it.

Not only was it's construction impressive, it can also take a pounding:

it has contained energy sufficient enough to destroy an entire galaxy and even energy sufficient enough to destroy one fifth of the Universe.

So if it can contain that much energy, it should be able to withstand denting making it more indestructible than Capt's shield. In fact Thor:

claimed that no power in the universe but the All-Father's could affect Mjolnir

It also has a variety of enchantments on on it. Here are some I think that may have been crucial to denting the shield:

Energy Absorption and Redirection Matter Manipulation

With those it would be possible to either redirect some previously tapped energy source and redirect it into the shield, or simply manipulate the matter in the shield to cause it to buckle (or transform a small part of the hammer into Arctic vibranium, adamantium's evil twin capable of damaging adamantium).

However, the hammer may not be that effective against true adamantium, or simply when thrown:

Thor throws Mjolnir at the robot Ultron with no effect in Avengers #68

It's debatable whether this is due the enchantments needing skin contact, or Thor just wasn't trying hard enough.

However this is mostly guesswork as the Odinforce is so powerful I'd say it border's on reality control.