If you're not eligible for premium-free Medicare Part A, you can sign up to buy it but need to do so as soon as you become eligible at age 65. Otherwise, if you sign up later, you'll pay a penalty equal to 10% of your monthly premium. You'll owe the penalty for twice the number of years you could've been covered by Medicare Part A but didn't sign up.

There's also a penalty for failing to sign up for Part B on time. The penalty equals 10% of your premium amount for each full 12-month period you were eligible to enroll but didn't. The penalty will be in effect for as long as you have Part B coverage.

You could also owe a penalty for part D, which is equal to 1% of the national base beneficiary premium multiplied by the number of months you were eligible but didn't have coverage. Since the national base beneficiary premium can change annually, the penalty for late enrollment can also change each year. The penalty you owe will be added onto your Part D premium for the whole time you're covered.

While there are some exceptions to penalties if you're eligible to sign up during a Special Enrollment Period, it's important to make sure you sign up when you're required to so you can avoid high costs.