One of the few remaining elegant wooden buildings in central Tokyo that survived allied bombing in the second world war is the Mitsubishi lodge. The Kaitokaku, set back from the main thoroughfares, its gardens overshadowed by skyscrapers, is a web of beautifully carved wood rooms where executives, bankers and clients who are part of the Mitsubishi keiretsu, or conglomerate network, socialise, with top officials from the phenomenally influential ministry of international trade and industry (MITI, more recently METI, ministry of economy, trade and industry), and discuss business privately – drinking sake until three or four in the morning. Deals are done, new technologies discussed and consensus reached on what is happening geopolitically to Japan’s markets and how it should react.

The closeness of the relationship between business, finance and state is a universe apart from anything we know in Britain. Critics argue that the keiretsu represents a destructive corporatism and an obstacle to the operation of free and open markets; admirers that it allows member companies to share risks, find synergies, lower financing costs and encourage long-termism – so informing Japan’s agile industrial policy and directly leading to Japan having dozens of world-beating companies and brands. By contrast, Britain is not at the races.

It will be in similar lodges around Tokyo that Brexit’s impact on Japan’s UK investments will now be discussed. In the 1980s this is where consensus was reached that Margaret Thatcher’s commitment to the European single market and customs union, along with her labour market reforms, offered a major opportunity for Japanese business. It would allow Britain to be the place where Japan could develop its famed “just-in-time” delivery system by sourcing production across Europe, without tariffs, regulatory and customs checks, and foster the recruitment and management of talent Europe-wide. They took her at her word, and the investment boom that has transformed the British car industry was born.

Today, a new Japanese consensus has formed. The Conservative party and its leaders cannot be trusted. They ignore warnings, break their word and do not understand business – personified by Old Etonians Boris Johnson and Jacob Rees-Mogg. Brexit is a first-order disaster, striking at the heart of how Japanese companies organise themselves as “lean manufacturers”. As Honda’s Patrick Keating, its European government affairs manager, briefed a meeting in Swindon in September, Brexit is likely to interrupt the just-in-time delivery of 2 million parts a day – a fifth of which come from EU suppliers. Those suppliers would have to fill out 60,000 customs declaration forms a year, he warned. One in five of its UK workforce are EU nationals. The world of tariff-free barriers – access to the EU’s free-trade agreements with other countries, and ability to move staff between countries promised by Thatcher – has evaporated in front of Honda’s eyes. Yes, it was operating below capacity, but within the EU Honda could have taken a long-term view and braved the downturn. Brexit forced the plant’s closure and the decision to produce in Japan.

The Honda keiretsu is allied to the Mitsubishi keiretsu which finances it and has members on its board: there is no doubt the decision will have been discussed until 4am over sake in the wood-carved rooms of the Kaitokaku. Its decision is part and parcel of the gathering consensus. In the last six months both Panasonic and Sony – both part of separate keiretsu conglomerates with their in-house banks and close relationships with METI - have moved their European headquarters from Britain to the EU mainland. The decision by Nissan, member of another keiretsu which includes Canon, Sapporo and Yamaha, not to build its new X-Trail model in Sunderland but at its Japanese production hub in Kyushu was inevitable.

Nissan has announced that the next-generation X-Trail will be made in Japan. Photograph: Christopher Furlong/Getty Images

British Conservative ministers are regarded by many Japanese as alien creatures locked in a 19th-century time warp. The letter from the international trade secretary, Liam Fox, and the foreign secretary, Jeremy Hunt, to METI, saying “time was of the essence” in concluding a free-trade agreement and urging “speed and flexibility” was seen as rude, impolitic and inviting Japan to disregard its interests – echoing an imperial mindset but with none of the accompanying economic clout. Japan intends to be ultra-tough in any trade deal it strikes with Britain.

It will want the same preferential tariffs on imported manufactures as it has with the EU, and intends to give as little away as possible on British financial services and food exports. If Fox wants better, best drink sake in the Kaitokaku, build trust and find something Japan needs – such as staying in the customs union and single market, or better still, inside the EU. It would never cross his mind. It is a purblind ignorance that extends to Justin Tomlinson, MP for North Swindon, claiming that Honda’s decision had nothing to do with Brexit.

The president of Honda Europe and the Japanese ambassador tried to soften the blow: Brexit was only one factor in the Swindon closure driven by wider trends in car manufacturing. It is important in Japanese culture to give face to people in public to avoid their embarrassment. But no one should be in any doubt. The Japanese car industry, Japan’s ambassador and its government have repeatedly warned of the implications of Brexit and been brushed aside. They are incandescent with anger – and feel betrayed.

Brexiters will argue that the impact of Japan’s withdrawal from a UK which last year created some 440,000 jobs can be handled easily. But that was in a UK economy of which Japanese investment was a part. The British economy is an ecosystem: high-paid jobs in car manufacturing in towns such as Swindon and Sunderland sustain other less well-paid jobs, while growth in business and financial service relies on company headquarters being here and not in Europe. Lose them, and apart from particular towns being hard hit, everyone is deflated. Of course, it would be better if there were more indigenous big British employers – but the same daffy rightwing thinking that produced Brexit has undermined British capitalism for decades. Japanese inward investment mitigated our own failures: and is now on its way out. Britain has many more bitter pills to swallow before this whole contemptible business is over.

• This article was amended on 27 February and 23 April 2019: after publication, Nomura contacted The Observer to state that it is not moving its European headquarters to the EU mainland, but rather that it is extending its lending licence in Paris and has been granted a securities trading licence in Frankfurt; and MITI’s name change to METI was added.

• Will Hutton is an Observer columnist