Research In Motion (RIMM) shares are up more than 164% over the past six months as hype surrounding the struggling vendor’s next-generation BlackBerry 10 smartphone platform peaks. While the increased anticipation may lead to success in the early days for RIM’s new BlackBerry Z10 and BlackBerry X10 smartphones, the company has a much steeper mountain to climb if it hopes to compete in the cutthroat smartphone market. RBC Capital Markets analyst Mark Sue is trying to keep a level head amid the buzz, and he paints a realistic picture of RIM’s prospects in 2013.

In a note to investors on Wednesday, Sue raised his price target on RIM shares to $19, up from $11, with a Sector Perform rating. He sees plenty of potential in BlackBerry 10, but he also warns clients that industry watchers may be getting sucked into the hype and are issuing overzealous estimates as a result.

Though some are calling for RIM to sell as many as 30 million handsets to current BlackBerry users, Sue says converting the “diehards” it just the beginning and convincing the bulk of RIM’s current installed base won’t be as easy as many believe.

“Our forecast calls for RIM to ship 500k BB10 units in the Feb-quarter and 10M in CY13,” the analyst wrote. “Heavy subsidies and promotions may be required to sustain initially positive trends. Approximately 10–20M of RIM’s 79M installed base may upgrade to BB10 over time, although migration to low-cost Android is likely to continue. BB10, which does not entail mandatory service fees, will increase RIM’s mix of hardware revenue over time. Sustainable hardware margins have been elusive for many vendors.”

Sue continued, noting that the stage is set for RIM’s early success with BlackBerry 10. He lists several things working in RIM’s favor, including 70,000 BlackBerry 10 apps available at launch, the fact that more than 1,600 businesses in North America are testing the platform, and the presence of several key points of differentiation in RIM’s new OS, such as BBM and the separation of work and personal profiles.

But solid early BlackBerry 10 performance is only the tip of the iceberg for RIM.

“The true measure of success will be not just units, but also sustainable profitability, and competition is cut-throat with Windows convinced they will be the third mobile OS vendor,” Sue wrote. “Heavy upfront spending may be required for the launch and we expect initially $500+ ASP and 10–15% gross margins on BB10 before volumes help margins.”