Avionics

For aircraft control systems, we don't speak of operating systems but of avionics, integrated avionics or computer airborne systems in general. And they are actually a combination of a multitude of independent or inter-dependent systems, for different functions (flight control, collision avoidance, weather, communications, blackboxes...).

Each controller is usually an independent module (hardware and software) for obvious security and safety reasons: they are critical control and monitoring systems and if one of them were to fail or get damaged, it's a rather big "inconvenience" for the people which are depending on the aircraft. Dependability takes all its meaning when you are in the flying machine. So usually it's

custom built for the purpose of their mission,

built to work independently and interface with other systems,

built to be fault-tolerant to its own failures AND to failures of the other systems it's interfaced to (because you don't want the pilot's sound system taking down the engine controls, for instance).

It's not a big computer running everything. If you think of it from the perspective of a military aircraft rather than a commercial one (though it's similar in this respect, the image might help): if a part gets shot at, you'd rather at least have some other parts be able to keep going (the part controlling the communications and safety systems might be interesting to keep alive...). Hence also the big bunch of buttons you see in jetliners, to keep track of the status of different systems.

They are usually either built as custom components operating their own system, or they are run and scheduled by a micro-kernel (in most cases, with support for real-time capabilities).

It depends across vendors and countries, obviously, but they usually at least need to follow rather strict sets of regulations, design requirements and protocol specifics, which allow for:

the control of their strict-compliance to security and safety standards,

the inter-communication with other systems (much better if that airplane you took off with in Reykjavik can "talk" to that ground-control equipment in Tokyo...)

Standardization Efforts

The DO-178B (revised in 1992) and its successor the DO-178C (revised in 2012) (and a bunch of associated documents) are an example of reference certifications for such compliance levels, and are recognized by the FAA (US), the EASA (EU), and Transport Canada, amongst others. Multiple other organizations are involved in the creation of such documents, like the EUROCAE.

Such airborne systems are usually bespoke software, but the following systems are known to be used in some airplanes:

WindRiver's VxWorks (see Aerospace & Defense uses),

QNX (actually not sure if QNX is used in airplanes, but it's used in ground control systems)

To give you a vague idea of elements built into an avionics system, this list of avionics acronyms points to some of them (with some overlap).

Notable Languages used in Commercial and Military Avionics

Apart from the usual suspects we know in the "mainstream" programming world, you'll come up some often referenced names like Ada, and some less known languages like the (dated and now "retired" since 2010) JOVIAL.

Related StackExchange Questions: