The situation is complicated. This is certainly something unprofessional that happened here. There can be unforeseen problems that needs to be handled, but communication is the key. No one should expect support from others in a workplace without providing a context first.

You mentioned your manager asked all of you to contribute towards the party fund out of the blue and without any prior communication regarding the "why" part.

Given that this is something that is happening for the first time, you should reach out to him and casually enquire about it in more details, something like:

Hey boss, I thought that was a company sponsored party and I did not receive any info about the contribution earlier. Was is supposed to be this way or is there something I'm missing?

If they acknowledge there is a problem, provide some details about what caused the problem and requests to contribute to cover up for this time, that can be seen as a genuine case of mistake and unintentional communication gap.

However, if you do not receive a response that explains the matters, I'd say do two things:

Pay the amount (Avoid the clash now 1) Polish up your resume and start looking for another job. (Avoid future clashes, too)

Here, someone did not do their job well (make proper arrangements for the event), and when failed to achieve the goal (get the amount settled), they passed on the responsibility (have the amount collected) to someone else, without even feeling the need to communicate about the problem, in first place. Clear sign of poor management, communication skills/ policies and decision-making, you don't want to work for a management/ superior like that.

Rewards and gifts (yes, throwing a party for "someone" is a gift, let alone asking for the employees to pay for it) should flow down the hierarchy, not upwards.

[1]: Remember, I'm not saying to put up with the behavior, but IRL, it makes little sense to create a squabble with a clearly nonsensical superior / management for a one time payment of $5.