The Jackson family patriarch had been battling terminal pancreatic cancer and was "not doing well" as of late, a source previously told PEOPLE

Joe Jackson, the music manager and father of 11, died early Wednesday morning, the family confirms to PEOPLE. He was 89.

“We want to thank you all for the support you have shown us as we grieve for the patriarch of our family. We mourn the loss of our father and celebrate the life of a man who sacrificed so much to give us the life and success we have today. Thank you for respecting our privacy during this time,” the family said in a statement.

Get push notifications with news, features and more.

The Jackson family patriarch had been battling terminal pancreatic cancer and was “not doing well” as of late, a source previously told PEOPLE.

Grandsons Randy Jackson Jr. and Taj Jackson acknowledged the news in tweets Wednesday afternoon, writing, “RIP to the king that made everything possible! I love you grandpa” and “Joe was loved by the entire family and our hearts are in pain. Let us grieve.”

Prince Michael Jackson, grandson of Joe and son of Michael, captioned a photo on Instagram. “This man is and always will be an example or sheer willpower and dedication. He didn’t choose the path that was the easiest but he choose the path that was best for his family,” he wrote.

“You taught me to take pride in the Jackson name and what it really means, you taught me dedication in the face of adversity and most of all you showed me strength and fearlessness. There is and never will be someone like you. Fly free and until we meet again The Hawk.”

Daughter LaToya added, “I will always love you! You gave us strength, you made us one of the most famous families in the world. I am extremely appreciative of that, I will never forget our moments together and how you told me how much you cared. #RIP Joe Jackson”

Image zoom Joe Jackson D Steinberg/BEI/REX/Shutterstock

On June 21, son Jermaine told the Daily Mail, “He’s very, very frail, he doesn’t have long. The family needs to be by his bedside — that’s our only intention in his final days.”

Just days later, Jackson appeared to share an emotional tweet hinting at his worsening health struggles. “I have seen more sunsets than I have left to see,” read the caption on a photo of Jackson, in his trademark hat, silhouetted against sun rays. “The sun rises when the time comes and whether you like it or not the sun sets when the time comes.”

However, his granddaughter Paris claimed that the message was in fact not written by the family head. “My grandfather did not tweet this,” she commented. “I’m not sure if he’s ever used this account.”

Over the years, Jackson — who was born on July 26, 1928, in Fountain Hill, Arkansas — had a series of health problems.

In November 2012, he recovered from a minor stroke at his Las Vegas home. Three years later, daughter Janet had to put rehearsals for her Unbreakable World Tour on hold when he suffered another stroke and heart arrhythmia in Brazil on his 87th birthday in July 2015.

In May 2016, he was hospitalized due to a high fever and in June 2017, he was released from the hospital after a car accident in Las Vegas.

Image zoom Joe Jackson in May 2014 Mike Marsland/WireImage

Joe is survived by his wife Katherine and their children Maureen, 68, Jackie, 67, Tito, 64, Jermaine, 63, La Toya, 62, Marlon, 61, Randy, 56, and Janet, 52.

Son Michael died from an apparent cardiac arrest at age 50 on June 25, 2009. Son Brandon died at birth in 1957.

Joe was also the father of Joh’Vonnie Jackson, 43, the daughter of Cheryl Terrell, whom he had a 25-year-long affair with. Katherine filed for divorce twice but never went through with the separation.

In 1963, Joe noticed his children’s musical talents after he discovered son Tito’s guitar talents. Soon after, Joe served as the manager for The Jackson Brothers, a band formed by sons Jackie, Tito and Jermaine. Then in 1965, Marlon and Michael joined the group and renamed themselves The Jackson 5 in 1966.

Anchored by a 7-year-old Michael, The Jackson 5 found immediate success, becoming the first American group to have four singles land the top spot on the Billboard Hot 100 chart. Soon after, Joe moved the entire family from Indiana to California.

In 1968, Joe signed a Motown contract—without reading it—that gave the company ironclad control over everything recorded by The Jackson 5, a move that cost the family millions. Michael fired his father as manager in 1983.

In addition, Joe was known as the stern patriarch who was physically and emotionally abusive.

In her 1991 memoir, La Toya wrote that he abused her and terrorized her brothers, often pointing guns at the kids to get their attention. During an Oprah special in 1993, Michael admitted his father had beaten him.

Joe’s estranged relationship with son Michael resulted in him being completely left out of the King of Pop’s will after his 2009 death. Katherine continues to receive more than $1 million annually from the Michael Jackson estate.

Image zoom Michael Ochs Archives/Getty

In a 2010 interview with Oprah Winfrey, Joe said, “I don’t [regret the beatings]. It kept them out of jail and kept them right.”

In 2014, Joe opened up about his health in an open letter to daughter Janet.

“When I suffered four strokes last year, and was in the hospital recovering, only two people in my family traveled all the way to see in the hospital. My granddaughter Brandi (Jackie’s daughter) and my baby girl Janet,” Joe wrote. “She sat right next to me as I lay in bed, spent time with me, talking about a lot together. It meant a lot to me. I lay there listening to her as she talked about how she was going to get married. I am happy for her and wish her nothing but the best! I never tell you this, but I am proud of you Janet.”