Johannesburg (CNN) -- In one Johannesburg prison there are 27 toddlers locked behind bars. The only crime they're guilty of is being born to a mother who broke the law and was jailed.

They are South Africa's littlest inmates, doing time with mom. For them, home is the cold, dark corridors and cells of the Johannesburg Female Correctional Facility.

Prisons around the world struggle with the question of whether female inmates should raise their babies behind bars. In South Africa, women are allowed to serve time with their children.

In the Johannesburg Female Correctional Facility, newborns are housed with their mothers in a hospital wing in a separate section of the facility.

But experts have concerns about what effect growing up in jail -- where the high walls, razor wire and isolation are intimidating even for an adult -- has on the babies.

Locking children away behind bars is not good for them, says Sisakele Zwane, who runs the overcrowded women's prison.

"It's not a nice thing. The correctional facility is not conducive to the growth of a human being, so for the children that are growing inside the correctional facility it is not a good thing at all," she said.

Authorities in South Africa face a dilemma. They know the mothers need to bond with their babies so they try to make the prisons child friendly.

But they're also aware that raising a child behind prison walls isn't ideal and insist that mothers send their children away -- to stay with family or go into foster care -- when they turn two.

Many inmates, though, have no choice but to keep their child with them. Some are foreigners, others are alone in the world and many just can't bear to send their baby away.

One inmate, Moipane Nkwana, was pregnant with her fifth child, a much longed for baby boy, when she was jailed for fraud.

"He is the only boy, with two sets of twins, girls ... I am in prison with a boy baby -- why now?" she said. "At the end of the day you don't have answers, you just have to take the challenges and move on."

Nkwana and her child cannot move on just yet. They're confined to their tiny cell at night and she says her son's life, like her own, is defined by the strict routines of prison life.

Mother and son have to do things such as line up every morning for roll call and obey the commands of the officers.

Nkwana says it's a depriving experience for her son. "The child doesn't understand the outside world," she said.

The mothers at the facility tell their toddlers about things they've never seen, such as cars, trains -- or men.

"They only see a nation of mothers, they don't know about fathers," Nkwana added.

The children are first separated from their mothers at six months. While their moms attend rehabilitation, the youngsters are sent to the prison day care.

The authorities feed, clothe and educate the children but they say these babies shouldn't be here in the first place.

"What we are saying is all these children have got fathers outside but the fathers don't pitch up, don't make themselves available to take them out of the correctional facility, so we now become their fathers inside here," said Zwane.

For many years in South Africa, children stayed with their incarcerated mothers until they were five. But prison authorities now say that's just too psychologically damaging for them.

Correctional services are now looking at changing the law so no child will have to grow up in jail in the future. There have been some suggestions that a new mother's sentence should be deferred until her baby is old enough for her to leave it behind.

Nkwana's sentence is nearly over but she knows that one day her son will ask why he was raised with strangers on the cold concrete floors of Johannesburg's largest prison.

She said: "He will ask you why? And you just can't answer that."