Millions of people living on the outskirts of Australia's major cities are being left behind by a lack of access to public transport which is impacting their quality of life, the nation's independent infrastructure advisor says.

Infrastructure Australia says close to half the population of the nation's five largest cities live in the outer suburbs, and have longer travel times and infrequent service compared to inner city residents.

"Across all five cities, a substantial number of people living in the outer suburbs do not have frequent public transport services within walking distance of their home," Infrastructure Australia Executive Director of Policy and Research, Peter Colacino says.