Story highlights Bae's sister says family is "really discouraged" by his move back to a labor camp

"I have not lost hope," Kenneth Bae says in a conversation with a Swedish diplomat

But he warns that the physical labor will probably put him back in the hospital

North Korea has held Bae, a Korean-American, since November 2012

Kenneth Bae , a Korean-American being held in North Korea, says he is worried about his health after authorities moved him back into a labor camp following a stay in a hospital.

"I know if I continue for the next several months here, I will probably be sent back to the hospital again," Bae says in a video of a conversation with a Swedish diplomat recorded Friday.

Footage of the conversation in the labor camp was released by Choson Sinbo, a pro-North Korean newspaper based in Japan that has been given access to Bae in the past.

Wearing a gray jacket with the prisoner number "103" marked on it, Bae tells the Swedish diplomat, Cecilia Anderberg, that he thinks his weight has already dropped as much as 10 pounds since he was transferred back to the camp a few weeks ago.

Bae's sister, Terri Chung, told CNN's "The Lead" that he looked worse than he did when he last appeared three weeks ago, in a hospital.

"I'm glad that he's holding strong, but I'm really concerned about his health," Chung said. "And the fact that he's been moved to the labor camp, we're really discouraged by that."

JUST WATCHED Campaign to free Bae from N. Korea Replay More Videos ... MUST WATCH Campaign to free Bae from N. Korea 08:28

JUST WATCHED Rodman: I'd trade places with Kenneth Bae Replay More Videos ... MUST WATCH Rodman: I'd trade places with Kenneth Bae 00:48

JUST WATCHED N. Korea moves American to labor camp Replay More Videos ... MUST WATCH N. Korea moves American to labor camp 02:03

The 45-year-old Bae, of Lynwood, Washington, was arrested in November 2012 in Rason, along North Korea's northeastern coast. Pyongyang sentenced him last year to 15 years of hard labor, accusing him of planning to bring down the government through religious activities. He is widely reported to have been carrying out Christian missionary work in North Korea.

U.S. civil rights leader Jesse Jackson has offered to go to North Korea to help get Bae released, a move the U.S. State Department says it supports. "We're just waiting and hoping for a positive response," Chung said.

"Every day we hold out hope that there is going to be some good dialogue between the two countries that will lead to Kenneth's release soon," she added.

In the video, he expresses hope that North Korea will allow a U.S. envoy to visit for talks about his case -- but those hopes appeared to have been dashed over the weekend.

A State Department official said Sunday that North Korea had rescinded its invitation to the envoy, Ambassador Robert King, without giving a reason.

Hours later, the North's state-run Korean Central News Agency reported that a former U.S. ambassador to South Korea, Donald Gregg, had arrived in Pyongyang.

The brief KCNA report Monday didn't state the purpose of the visit by Gregg, the chairman of the Pacific Century Institute, a U.S. nonprofit group that aims to promote education, dialogue and research in the Pacific region. But U.S. State Department spokeswoman Marie Harf said Gregg was traveling with other former diplomats on a trip to "build bridges," and his visit was unrelated to Bae's case.

In a written statement earlier Monday, Chung said the family was "saddened to hear" that King's invitation had been rescinded. But she added, "While we reel from this heartbreaking news, we are encouraged by the growing chorus of advocates asking for Kenneth to be released."

And Jackson told CNN that he has written to North Korean leader Kim Jong Un to request a face-to-face meeting "to make a humanitarian plea to him."

"He is repentant and has been contrite, and we hope for forgiveness," Jackson said of Bae.

In Washington, Harf told reporters that the State Department "certainly" supports Jackson's effort.

'I have not lost hope'

Bae operated a China-based company specializing in tours of North Korea, according to family members, who have described him as a devout Christian.

He was transferred to a hospital last year after his health deteriorated. But last week the United States said he had been moved back to a labor camp, a development his family described as "devastating."

In the video, Bae asks the Swedish diplomat to tell his family that "I have not lost hope and have not given up anything."

But says he is concerned that if his situation isn't resolved soon, it could "drag on" for months longer. He notes that annual U.S.-South Korean military drills due to start later this month may deepen tensions in the region, as they did last year.

State Department spokeswoman Jen Psaki on Sunday expressed disappointment that King's visit was called off and noted North Korea had said it wouldn't use Bae as a "political bargaining chip." It is the second time North Korea has canceled a planned visit by King.

Psaki said that the joint military exercises are "in no way linked to Mr. Bae's case."

North Korea has been urging the South not to take part in the drills -- a call that Seoul and Washington have rejected. Jackson said he hoped a visit would break "that cycle of terror and fear and irritation" between the two countries.

Life in the camp

In the conversation Friday, Bae discusses details of his health problems, as well as the minutiae of life in the labor camp.

He says he is suffering from back pain and neck pain, making the eight hours of manual labor he does each day "very difficult."

"I've been working with my hands a lot," Bae tells the diplomat. "My hands all got numb and sore. I have some cuts."

But he says that he remains "strong mentally and spiritually, and I am trying to stay strong emotionally as well."

Bae tells the diplomat that he has access to books and television at the camp and that the staff there treat him "very fairly."

The TV antenna stopped working for a couple of weeks recently, he says, allowing him to spend "more time with the Lord, with the Bible."

"That was actually a pretty good time for me," Bae says.

Sweden represents U.S. interests in North Korea because the United States has no diplomatic presence in the secretive state.

"We again call on the (Democratic People's Republic of Korea) to grant Bae special amnesty and immediate release as a humanitarian gesture so he may reunite with his family and seek medical care," Psaki said Sunday. "We will continue to work actively to secure Mr. Bae's release."