Astronomy is in the middle of a data revolution, a time of enormous discovery. Humans have been looking up at the stars for thousands of years, but modern telescopes and computers are rapidly accelerating our understanding. We know far more than we did even 20 years ago, and we are now much closer to answering questions about whether life exists elsewhere in the cosmos, how our planet came to be here, our cosmic origins and eventual fate.

On a clear night the stars twinkle in the sky, and often we can see the bright planets of our own solar system in orbit, like us, around our sun. Until the early 1990s, we had no idea whether there were any other planets around the stars in the sky. Astronomers suspected so, but had no way to prove it. We have in the past decade found thousands in orbit around foreign stars, and have a better idea of how common they are. A sizable fraction of stars probably have their own planets, their own worlds carried around them in the most fantastically diverse solar systems.

The Pistol star, in an infrared image taken by the Hubble space telescope, is located near the centre of the Milky Way. Photograph: UCLA/Nasa/HO/REUTERS

Look up at the stars, though, and you can’t see any of their planets. They are tiny and don’t make much light of their own. Even with powerful telescopes it is hard to see a planet directly, as the light from its parent star is blindingly bright in comparison. Instead, astronomers have devised ingenious methods to find them. One is the “wobbling” effect: a planet makes its parent star wobble just a little as it orbits around it, the star feeling a subtle pull of gravity from the smaller planet. In 1995 the Swiss astronomers Michel Mayor and Didier Queloz used the Haute-Provence Observatory telescope in France to pinpoint a star, 51 Pegasi b, that was wobbling just as expected. It revealed a planet, similar in size to Jupiter but so close to its star that it takes only four days to orbit around it, far quicker of course than Earth or any of the planets in our solar system.

Astronomers have since found a huge haul of planets using Nasa’s Kepler satellite, which launched in 2009 and ended its spectacularly successful mission late in 2018. Kepler was designed to spot planets crossing in front of their parent stars, looking for the slight dimming effect of the planet blocking some of the starlight. Kepler has found thousands of planets this way, including many whose complete orbit around their star takes just hours or days. Some are so close to their star that their surface must be entirely molten lava. It has shown us solar systems with multiple planets, and some planets that orbit around two stars – “Tatooine” planets, named after Luke Skywalker’s fictional home. Some planets are larger than Jupiter, some are as small as our moon. Most were found by professional astronomers, but amateurs have found them too, using pattern recognition to mine online data from the Kepler mission and catch planets that automated searches had missed. We now think there are more planets than stars in the sky.

Neutron stars are nature’s weirdest. They are incredibly dense – a spoonful of neutron star weighs as much as a mountain

We want to find out how likely it is that a habitable planet like Earth exists and, even better, to find one. In the elusive “Goldilocks” zone, the conditions must be just right: not too hot and not too cold. Astronomers have already found a few dozen planets like this, of which at least three are in the Trappist I solar system, found in 2015 around one of our nearest stars. Trappist I has seven planets, all similar in size to Earth, and at least one has a liquid water ocean. Studying these planets, and some of the hundreds of millions of potentially habitable planets in our Milky Way galaxy is the goal of ambitious new projects. Nasa’s Transiting Exoplanet Survey Satellite (Tess), launched in 2018, is expected to find more than 20,000 new planets. The most interesting will be examined in greater detail by the James Webb Space telescope, the Hubble space telescope’s successor, due to fly in the 2020s.

The stars in the sky are hosts to this wealth of solar systems, but they also have a vital role to play as nature’s great factories. Consider gold: where did it come from? Astronomers have long wondered how gold came to be part of Earth’s ingredients. We know that it is stars themselves that manufacture what our own bodies are made of, including mostly carbon and oxygen. The British-American astronomer Cecilia Payne-Gaposchkin first worked out in the 1920s that stars are themselves mostly just hydrogen and helium gas. The intense heat at the heart of stars can fuse together these tiny atoms into slightly larger ones, which are then ejected into space when the stars explode at the end of their lives. Those recycled remnants will later be present at the birth of new stars like our sun, and will end up making new planets like our own. But the alchemy of making gold has long been a mystery. Gold has much larger atoms than carbon and oxygen, and can’t be made in the core of a star. The explosions of individual stars could create some of it, but not enough.

Jocelyn Bell Burnell at the Mullard Radio Astronomy Observatory at Cambridge University, in 1968. Photograph: Daily Herald Archive/SSPL/Getty Images

In 2017 astronomers concluded that another way gold is generated is during intense collisions of pairs of neutron stars. Neutron stars are nature’s weirdest. They are incredibly dense, the result of squashing a star larger than our sun into a space just a few miles across, after it has run out of the fuel that kept it burning bright. A spoonful of neutron star weighs as much as a mountain. We can’t see them with regular telescopes, but they send out radio waves. The stars spin around and, if oriented right, their beam of radio waves sweeps past Earth regularly, like a lighthouse beacon. The astronomer Jocelyn Bell Burnell first discovered one of these stars in November 1967 using a radio telescope she built (with Anthony Hewish) as a PhD student at the University of Cambridge. The pulses of radio waves she saw from the star were so regular, arriving every second, that they initially named it LGM-1 for “Little Green Men”, half-jokingly considering that it might be a signal from aliens.

When a pair of these neutron stars orbit around each other, their pull of gravity is so strong that they actually deform the fabric of space-time itself. A gravitational wave ripples outwards, subtly shrinking and then expanding space and everything in it, over and over, as it passes on by. On 17 August 2017 the Laser Interferometer Gravitational Wave observatory in the US (Ligo), and the Virgo detector in Italy felt a burst of these waves passing through Earth. Using lasers to measure the precise length of two 4km-long tubes positioned at right angles, they directly measured the tiny squeezing and stretching of the tubes. They were sensing two neutron stars for the first time, catching the last two minutes of their life as they spun thousands of times around each other before colliding.

The Ligo and Virgo teams immediately alerted astronomers around the world, so that they could search for the aftermath. It was still daytime in Hawaii and Chile, home to many of the largest telescopes, so they had to wait until nightfall. Astronomer Ryan Foley and his team from the University of California in Santa Cruz were the first to locate a burst of intense light coming from the collision, using the Swope telescope at Las Campanas Observatory in Chile. They in turn alerted the community so that other telescopes could swing to take a closer look. In the end it was seen by 70 observatories on all seven continents and from space. What they had found was a kilonova and by studying the explosion, astronomers could tell that it had produced huge amounts of gold and platinum, much more than the entire weight of Earth. That particular collision happened too far away to have produced the gold on your finger, but here at last was decisive proof that closer to home collisions like this could have created Earth’s gold.

The stars, source of heat and light and of our very bodies, give an impression of permanence. There they are, night after night. We do, though, already know that they will change. Their positions in the sky will gradually shift, as our own solar system moves around the Milky Way galaxy, our larger home to billions of stars. And they will eventually stop burning bright, and new ones will be born. Within astronomy, my particular expertise is cosmology, in which I ask questions about the life story of the whole of the universe. As a professor at Princeton University I also teach undergraduate students, and I train junior scientists to ask, and answer, their own questions about the cosmos.

By the late 20th century astronomers had begun to consider a big change on the horizon. If we look beyond our own Milky Way galaxy, we find a sea of galaxies, each a collection of billions of stars, strewn throughout space. A century ago, the astronomers Edwin Hubble and Georges Lemaître showed that the galaxies are, on average, all moving apart from each other. Space itself is growing. But the pull of gravity of all the galaxies towards each other should tend to slow down the growth of space, just as a ball thrown in the air comes to a halt and then falls back down again. So astronomers turned to wondering whether space would one day stop growing altogether, and then start to shrink. Billions of years in the future, might all the galaxies come crashing back towards each other, invading our local cocoon of stars and marking the end of the universe as we know it?

The key to answering this lay in measuring how fast space is growing and comparing it to its growth in the past. You can of course see into the past by looking far out into space, at places from which light set off a long time ago. Using a set of telescopes in Chile, Hawaii and the Canary Islands, rival teams of astronomers made new measurements of distant galaxies, using extremely bright exploding stars as cosmic markers. The teams, led by Brian Schmidt at Mount Stromlo observatory in Australia, and Saul Perlmutter at the University of California in Berkeley, measured galaxies whose light set off as long as 7bn years ago. In 1998 both announced the same surprising discovery, which was that space was not slowing down at all. Instead, galaxies appeared to be moving apart from each other faster and faster. This was as peculiar as if you threw a ball in the air, and it sped up and away from your hand into the sky. By the early 2000s other measurements of the universe were telling the same story, and over the past decade the evidence has become incontrovertible. We still don’t yet know why this is happening. Astronomers refer to whatever is causing this as “dark energy”, a form of energy we have not yet identified. It could be the energy of empty space itself, or something entirely new. It is a mystery waiting to be solved.

This new finding means that we no longer expect all the galaxies in space to come crashing back towards us. We are probably safe from that fate. But our own Milky Way is still on a collision course with at least one other galaxy. We are flying towards Andromeda, the nearest major galaxy to the Milky Way, at a speed of more than 50 miles per second, and there is no turning back. A few billion years from now we will unavoidably collide and mix our stars together. Any living beings in our solar system by then will see a spectacular change in the night sky when it happens, but the gaps between stars is so large that it is unlikely any will physically collide. Our sun will simply find itself part of a larger home.

This future view of our cosmos, our revolutionary discoveries of new worlds around stars, and our understanding of how the inner workings of stars make the fabric of our lives, are just some of this century’s exciting discoveries. A common thread is that major advances in astronomy take teams of scientists working together, using telescopes around the world and in space, coupled with powerful computers. The questions we are asking cannot, for the most part, be solved by people working alone. As a result, the community is international. Meet an astronomer anywhere in the world, and most likely they are asking the same questions about the existence of life elsewhere, our place in space and the cosmic story of how we came to be here. And with many magnificent new telescopes being readied for the coming decade, we can expect together to take further great strides in our understanding of the night skies, and the wider universe, in the coming 20 years. We know, based on past experience, that the most exciting discoveries will probably be the ones we have been least expecting.

• Jo Dunkley’s Our Universe: An Astronomer’s Guide is published by Pelican on 31 January.