Baltimore, Maryland (CNN) He was a quiet man who once stood watch on his front porch, just three blocks away from where a riot erupted in West Baltimore this week.

We called him "Mr. Shields" because no one dared use his first name. He'd step onto his porch at night in plaid shorts and black knit dress socks to watch the Baltimore Orioles play on his portable television set.

He was a steelworker, but he looked debonair: thin mustache always trimmed; wavy salt-and-pepper hair touched up with pomade; cocoa brown skin. He sat like a sentry, watching not just the games but the neighborhood as well.

I knew Mr. Shields' routine because I was his neighbor. I grew up in the West Baltimore community that was rocked this week by protests over the death of a young black man in police custody.

Mr. Shields with his daughters Sandra, left, Linda and Elaine, right, in an undated family photo.

It's surreal to see your old neighborhood go up in flames as commentators try to explain the rage with various complex racial and legal theories. But when I returned to my home this week, the rage made sense to me. There were no more Mr. Shields -- the older black men were gone.

I asked 28-year-old Zachary Lewis about the absence of older men. He stood by a makeshift memorial placed at the spot where Freddie Gray, the man whose death ignited the riots, was arrested.

"This is old here," he said, pointing to himself. "There ain't no more 'Old Heads' anymore, where you been? They got big numbers or they in pine boxes." In street syntax, that meant long prison sentences or death.

We hear about the absence of black men from families, but what happens when they disappear from an entire community? West Baltimore delivered the answer to that question this week.

It's no accident that one of the most enduring images from the riot was a young mother spanking her son as she dragged him away from the protests. Where were the men in his life?

As I walked through my old streets, it was filled with nothing but black young women, children and teenage boys. It was as if an alien spaceship had come in the night and spirited all the older black men away.

Castaways waiting for rescue

I've read and written about big issues like the mass incarceration of black men for nonviolent drug offenses -- what some call "The New Jim Crow." To see it in person, though, is spooky. I felt like "The Lord of the Flies" had taken over my old neighborhood.

JUST WATCHED 'Lord of the Flies' comes to Baltimore Replay More Videos ... MUST WATCH 'Lord of the Flies' comes to Baltimore 04:35

"The Lord of the Flies" was a novel written in 1954 by the English author William Golding. It describes what happens to a group of upper-class English schoolboys when their plane crash-lands on a deserted island and all the adults are killed. The kids try to build a society of their own, but with no adult guidance, they descend into tribalism and savagery.

William Raspberry, a Pulitzer Prize-winning writer for the Washington Post, once invoked the book's title in a column to describe what was happening to young black men in inner cities across America. He said that without the civilizing influence of older men to guide them, young black men never develop an internal moral compass. They become castaways. I read Raspberry's essay after college and kept it for years. It spoke so well to what I saw in the 1980s when the crack epidemic first hit my neighborhood.

I heard Raspberry's voice again this week when I talked to a 27-year-old black man named Juan Grant. He knew Gray, whose death in police custody lit the fuse in Baltimore. Grant stood no more than a foot from me, but as he talked, he yelled at me in frustration, spittle coming from his mouth. He said Gray's death had convinced him and his friends to stop "ripping and running" the streets. They wanted boys to respect them as men.

But they didn't know how to get that respect because their fathers had never been around. He described their dilemma with a bitter laugh:

"It's men learning on the job trying to teach young men how to be men."

Raspberry wrote his column 28 years ago. Now there are even more castaways like Grant in West Baltimore. Yet here's the twist: They don't just feel abandoned by indifferent white people; many feel ignored by the city's black political leaders.

Baltimore Mayor Stephanie Rawlings-Blake is black, but I found nothing but disdain for her in West Baltimore. People kept complaining that she called protesters "thugs."

Before the Inner Harbor in Baltimore became a haven for tourists, it was a haven for a burgeoning black middle class in the city.

"She turned on her own people, calling us thugs," a 16-year-old high school student named Malik said as he waited at a bus stop next to Mondawmin Mall, a flashpoint for the riots. "Pretty sure she ain't perfect. She made some mistakes in her life. I'm pretty sure she did."

He doesn't think any of the city's leaders care about him.

"They talk about 'We the future,' but they killing us," he said.

Now this is the part of the discussion about Baltimore that some conservatives tend to love. Their refrain: It's all about individual behavior; there's a culture of poverty that Big Government programs won't help; Oh God, not Al Sharpton again; just pull yourself up by your bootstraps.

My own experiences tell me it's not all about individual effort.

What is a working man?

Choices, someone once said, are constrained by circumstances. And the circumstances that drove young black men like Grant to the streets this week are getting grimmer.

Take Mr. Shields, for example. The reason I saw so many working-class men like Mr. Shields in my neighborhood was because there were blue-collar jobs for them. Before the Inner Harbor in Baltimore became a haven for tourists, it was a haven for a burgeoning black middle class in my city.

Mr. Shields worked as a steel rigger at Sparrows Point in the inner harbor. Other black men worked at the Domino Sugar plant. My father was a merchant marine who sailed out of the harbor. These were well-paying jobs with strong unions that fought for good benefits and united black, brown and white working-class people.

The old Domino Sugar factory on the Baltimore harbor. Men like Walter Boyd raised families working there.

They helped men like Walter Boyd, 76, who sat on his front porch in West Baltimore this week like he was a reincarnation of Mr. Shields -- an impassive Sphinx surveying his domain. He was one of the few older black men I saw around. He had a box of chicken wings attached to his walker along with ice water. Boyd had raised three children working at Domino Sugar.

"Best job I ever had," he said. "You didn't get fired. You fired yourself. As long as you came to work and worked, you had a job. It was hard work but I had it made because I knew how to work."

Boyd's son, Robert, had just stopped by to cut his father's hair. He chuckled at his father's reference to hard work. Growing up, he said, his father kept him busy to keep him out of trouble. He'd take him to the country in the summer to work in the tobacco fields. He remembers watching his father plow a field one day, sweat pouring down his face, when another man turned to him and said with admiration: "That's a working man."

"There was something about the way he said that that let me know that's the way you supposed to be if you wanted respect," said Robert Boyd, who is a truck driver and pastor of the Beacon of Truth Church and Ministries in West Baltimore.

Yet boys don't respect men who don't have jobs. And many of those blue-collar jobs that built the black middle class in Baltimore are gone. Even the neighborhood businesses that I remember from youth -- an ice cream factory and a milk company behind my house -- were shuttered when I returned.

Unlike Walter Boyd, the old men I did see in my neighborhood this week were broken-down, unshaven. I thought to myself: If you want to destroy a people, first break their men.

"Now we as men are fearful when we walk through a group of boys," Robert Boyd said. "When we were boys, when we walked through a group of men, we felt secure. Something is wrong."

Taking the city away from us

Something else was missing when I returned: places for kids to play or meet the men who could mentor them.

Baltimore is a sports-crazy town. I grew up playing Little League baseball, running around the track at the high school across the street from my home, and playing tennis at public courts scattered through West Baltimore. There were public swimming pools, pickup basketball games, and plenty of recreation centers. On some days, I barely ate because I spent so much time outside playing sports.

Yet when I returned to my old playing fields, they were overgrown with weeds or barred with locked gates. I heard the same story from residents. The city had closed the pools, removed the basketball goals and, as recently as 2013, closed 20 recreation centers. I didn't see any kids playing baseball or football in the streets.

"They've taken the city away from us. We have nowhere to go and nothing to do," says Grant, the young man who wants to be a role model.

Men like the author's father were merchant marines and sailed from Baltimore's harbor.

The sports venues weren't just for the kids; they were for adults. It's where men mentored kids by becoming their coaches. The tracks and pools were places where families gathered. The school's playing fields were open to everyone in the community.

I practically lived on the playing fields at Frederick Douglass High School, which became a focal point for the riots. When I talked to Walter Boyd and his son, I did so across the street from Douglass' track, which was ringed with locked gates.

"I used to do my walking there," Robert Boyd said, pointing to the track. "Not just I, but older cats and younger cats would just walk. That's when you saw community -- younger, older people. You see people and say, 'How you doing.' They don't want you on the track now."

The youth aren't missing just recreation centers and tracks; the jobs programs are gone as well.

When I grew up in West Baltimore in the late 1970s and early 1980s, virtually all of my friends worked. The city offered various jobs programs for youths -- Summer Corps, Youth Corps, Manpower. Some jobs were as simple as sweeping the streets, but we didn't mind. It was like a rite of passage into adulthood. You didn't have to ask your parents for money. I still remember the envy I felt when my friends took their first Summer Corps checks to Mondawmin Mall to buy new tennis shoes.

I hear people talk about welfare queens and the "culture of poverty." But most of the kids I grew up with weren't even content to join a jobs program. They hustled for other work. One of the most coveted jobs was riding on the milk trucks during their morning deliveries. At sunrise virtually every day, a crowd of boys would gather outside the loading dock at the Cloverdale milk company. They stood around like the day laborers who hang out today around Home Depot. They wanted a milk driver to stop and say hop on. They'd help deliver the milk, and the driver would give them a couple extra bucks.

I still remember the rejection one morning when I woke up early and joined that crowd. One by one I saw all my friends picked up until I was the only one left. Nobody stopped for me. I was too skinny to pick up a milk crate. I went home and threw myself on my bed in despair. I would never be cool like my friends.

Photos: Cleaning up after Baltimore riots Photos: Cleaning up after Baltimore riots People help clean up a CVS pharmacy in Baltimore on Tuesday, April 28, the morning after riots broke out during protests on Monday, April 27. The unrest followed the funeral for Freddie Gray , who died of a severe spinal cord injury he got while in police custody. There were more than 230 arrests, 144 vehicle fires and 15 structure fires, a city official said. Hide Caption 1 of 13 Photos: Cleaning up after Baltimore riots Members of the community clean up debris on April 28 from the CVS that was burned and looted in the riots. Hide Caption 2 of 13 Photos: Cleaning up after Baltimore riots A neighborhood cleanup crew works to clear trash from the streets on April 28. Hide Caption 3 of 13 Photos: Cleaning up after Baltimore riots Caroline Byrd helps with cleanup efforts on April 28. Hide Caption 4 of 13 Photos: Cleaning up after Baltimore riots Volunteers help clear trash from a looted business on April 28. Hide Caption 5 of 13 Photos: Cleaning up after Baltimore riots A man sweeps the street as law enforcement officers stand guard on April 28. Hide Caption 6 of 13 Photos: Cleaning up after Baltimore riots Baltimore firefighters inspect a burned store on April 28. Hide Caption 7 of 13 Photos: Cleaning up after Baltimore riots People helping clean debris are seen in the reflection of a broken window on April 28. Hide Caption 8 of 13 Photos: Cleaning up after Baltimore riots Jason Park, left, and business owner Sung Kang survey the damage to his store on April 28. Hide Caption 9 of 13 Photos: Cleaning up after Baltimore riots Men wearing work gloves help clear debris on April 28. Hide Caption 10 of 13 Photos: Cleaning up after Baltimore riots Kailah Johnson, 5, joins her mother in a neighborhood cleanup on April 28. Schools were closed across the city. Hide Caption 11 of 13 Photos: Cleaning up after Baltimore riots Volunteers help clear the wreckage from damaged stores on April 28. Hide Caption 12 of 13 Photos: Cleaning up after Baltimore riots Employee Sam Wirtz, left, surveys the damage to his store that came during the unrest on Monday, April 27. Hide Caption 13 of 13

My interest in journalism also was nurtured by these jobs programs. I interned at the Baltimore Sun and Afro-American newspapers while I was in high school. I participated in journalism getaways for promising inner-city students. I couldn't afford any of it, but if you're reading this now, it's because somebody somewhere was willing to pay money to give me a chance.

Today, few of those programs exist. The Rev. Jamal Bryant, a popular Baptist minister in Baltimore, said the city has even closed a quarter of its public libraries.

"All of those programs are housed in the Smithsonian Institution," he says of the youth jobs programs. "They are no longer in evidence or thriving today."

Yet there is one institution the city seems to find money to invest in, some residents say: law enforcement. Funding for public schools, libraries, jobs programs and recreation centers may lag, but the budget for jails and police never seems to run dry, Walter Boyd and others say.

Some wonder if it's deliberate.

"If you don't invest in them now, you're just going to have to build more prisons," Boyd says about kids in West Baltimore. "And that just seems like that's what the plan is. They won't educate you. But they'll incarcerate you in a minute."

A bittersweet reunion

I ended my return by going back to the house where I grew up. I rang the doorbell, but a guy washing his car on the street told me the old woman who lived there wouldn't answer the door because she was "skittish." Bars seemed to cover every window; other homes were boarded up, and those that weren't looked so dilapidated that it seemed as if the residents didn't care anymore.

And they don't, because so few are owners now.

I ran into one person who was still there from my childhood. I knocked on his door and a big smile flashed across his face. He had not seen me since high school, but he remembered. We all called him "Herb." He was one of the few homeowners left.

We sat down on his porch and talked about old times. He said nobody sat on the porches and talked to each other anymore. Of the 38 homes on our block, only seven were owned by their occupants. When his house was recently burglarized, he said it took three calls to 911 and 55 minutes for the police to show up.

"I could be mutilated and lying on the street," he said, "and nobody would help or call the police."

I said goodbye and left. As I got in my car, I looked at him standing at his door, still smiling as he waved at me. I also looked at Mr. Shields' old porch as I drove away. The paint was peeling and the front looked disheveled. He never would have allowed that.

This was my home. This was my family. These were my friends. But they were ghosts now. There were few men looking out for the neighborhood any longer.

What's left are boys trying to figure out how to be men -- and how to avoid getting "big numbers" or ending up in "pine boxes."