Almost 150 artefacts lent to a museum set up to tell the story of Britain's colonial past may be missing, it has emerged, with some of them having been sold without their owners' permission.

Trustees of the British Empire and Commonwealth Museum in Bristol, which has now closed, are in talks with about six of the owners about compensation.

Among them is Lord Caldecote, who said he was shocked to find that a 19th-century maritime painting his family had lent to the museum had been sold at auction.

Much of the museum's collection has been handed to Bristol city council, which is carrying out a full audit. Police have investigated the loss of objects but have not made arrests or brought any charges.

An investigation by BBC's Inside Out West programme, scheduled to be broadcast on Monday, claims that 144 objects belonging to eight lenders remain missing. They include the oil painting of an East India Company ship, Dunira, by the sailor-turned-artist Thomas Buttersworth.

Caldecote told the Guardian that his late father, an engineer and industrialist, had lent the painting to the museum. After his father's death, he asked for the painting to be returned.

"I decided I would like the picture back. It turned out the museum had sold the picture through Christie's. I don't suppose we'll be able to get it back again."

Caldecote said the picture had sentimental value because an ancestor had captained the ship, part of the East India Company's fleet, and it had been a gift to him. "It was a shock when I found out the painting had gone," he said.

The painting was sold by Christie's to the government of Madeira for £61,250 in 2008. The island can be seen in the background of the picture. Neither Christie's nor the Madeirans realised that there was any issue with the ownership of the painting.

There is an ongoing dispute between the board of trustees of the museum and its former director, Gareth Griffiths, over missing artefacts. There is no suggestion that anyone has made personal profit from any sales.

The board has criticised Griffiths but he insists the care and security of the collection was the trustees' responsibility. He said: "I never benefited from any sales of material and will regard any such inference as actionable."

In March last year, sources close to the museum told the Guardian only "three or four" items were missing.

Neil Cossons, who took over as chairman of the board of trustees when the museum closed to the public in 2008, conceded the BBC's figure of 144 may be correct, but he could not be sure.

He said this was in part because of confusion over the way some objects were counted: whether, for example, a group of African spears was in fact one object or several.

He said it was fair to say that "dozens" of items were unaccounted for but also admitted that it may never be known exactly how many objects went missing. "It's hard to be mathematically precise," he said.

The museum opened in 2002 in Isambard Kingdom Brunel's original Temple Meads station and won a string of awards. Bristol was chosen because of its historic connection with empire through voyages of exploration, trade – including slavery – piracy, shipbuilding and the railways.

However, it struggled to attract enough visitors to make it financially viable and a plan was hatched to move the museum to London. The plan fell through after the financial crash.

Cossons, a former chairman of English Heritage, said the museum had been a "good notion" but had not worked in the tough economic climate. He said a large number of objects had been returned to their owners.

Julie Finch, head of museums, galleries and archives in Bristol, said a full audit, which had not been done before, was under way. The city council has begun using images and films from the museum to complement its collections. "The future looks bright for the collection," she added.

BBC Inside Out is on BBC1 in the west on Monday at 7.30pm.