Both views you've heard are somewhat right. Singapore was already an important port by the time of its independence, yet its growth accelerated so quickly after independence that the Singapore of forty years ago probably does seem like a backwater:

According to official statistics released since 1960, the economic indicators of Singapore show a sustained and relatively rapid economic growth over this period. For the period 1960–2000, real GDP (1990 prices) rose at an average annual rate of 7.7 percent. With population growth at 2.2 percent, real per capita GDP increased by 5.5 per- cent on average each year. In fact, real per capita GDP has increased 9.7 times within 40 years. Out of 107 countries, Singapore registered the highest growth performance during 1960–2000 (source)

Doubts about Singapore's viability as an independent nation probably had more to do with its size rather than level of development. Here is a table from Sugimoto's Economic Growth of Singapore in the Twentieth Century:

Those numbers are not per capita statistics. They show that the Malayan economy was almost three times larger than Singapore's at the time Singapore was expelled. Obviously, though, things worked out for Singapore. Using Maddison's data, we see that Singapore started out with a GDP per capita 1.5x as large as Malaysia. Singapore has only widened the gap to almost 3x since. Here's a table from Maddison, showing the ratio of Singapore's GDPpc to that of Malaysia, the UK, and the US:

So again, both views you've heard are somewhat correct. Singapore was always an important port in its region, and thus at independence it was wealthier than its nearest neighbors--but in 1965 it was a poor backwater compared to the U.S. and U.K: Singapore was 1/5 and 1/4 as wealthy (per capita) respectively. By 1993 it was as wealthy (per capita) as the U.K. and by 2010 it was approaching American levels.

PS - @KennyLJ's answer relies on the highly respected data series put together by Angus Maddison. Maddison's statistics post-independence are probably derived from official statistics (which begun in 1960, a year after Singapore achieved self-governance). So I think KennyLJ's answer more than qualifies for the bounty. The above is just for context.