Ever since we first dreamed of life on other planets, science-fiction writers have explored the potential consequences of an interstellar culture clash. Typically, they depict alien species as either hostile and seeking to conquer Earth – as in H G Wells’ War of the Worlds – or friendly and misunderstood, as in the blockbuster film E.T. It is the former scenario that drives the fictional events of Cixin Liu’s The Three-Body Problem.

This book – the first in a trilogy that is already a bestseller in Liu’s native China, and has now been translated into English by Ken Liu – offers a refreshing take on the well-worn “hostile aliens” trope, creating an intricate, suspenseful and multi-layered novel peppered with memorable characters marked by tragedy. Among these characters is an astrophysicist called Ye Wenjie who, as a young woman during China’s violent Cultural Revolution, witnesses the brutal murder of her physics professor father at the hands of his former students during a public rally, or “struggle session”. His crime: refusing to denounce modern physics theories deemed ideologically incompatible with the Communist regime.

Ye Wenjie is more fortunate than her father. She survives the purge, and as part of her cultural redemption, she is recruited to work on a secret military project to transmit radio signals into space in search of intelligent life – the Chinese equivalent of SETI. Those signals eventually reach an alien race on the brink of extinction, who set forth to conquer Earth.

Fast forward 40 years. New and wildly inconsistent results from particle accelerators are baffling physicists, including many members of the Frontiers of Science, an elite cadre of influential scholars. Several members have begun committing suicide, purportedly devastated by the possibility that invariance might not hold across the universe. “All the evidence points to a single conclusion: physics has never existed and never will exist,” one despairing string theorist writes in her suicide note.

Meanwhile, a nanomaterials researcher named Wang Miao is recruited by the military and asked to infiltrate the Frontiers of Science as part of a top-secret programme to prepare the planet for interstellar warfare. In the course of his investigations, Wang soon finds himself enthralled by a strange online immersive video game called Three-Body.

The name of the game alludes to a problem outlined in Isaac Newton’s Principia in 1687: predict the movement of three moving objects, taking into account the effects of their mutual gravitational attraction. It is a relatively simple matter to calculate the movement of two objects, such as the Earth and the Sun, but adding a third body, like the Moon, makes for a much more difficult calculation.

Three-Body players attempt to solve the three-body problem in order to predict the strange weather patterns on a virtual world with three suns. This world vacillates between chaotic and stable eras, and civilization can thrive only during stable eras. Players propose solutions and then see how their hypothesis plays out in the ensuing simulation – how long can their virtual civilization survive?

It is not a frivolous exercise. The game turns out to be part of an elaborate campaign by the same aliens who received those radio signals 40 years ago – inhabitants of an actual planet with three suns – to recruit possible allies from the disillusioned ranks of mankind. Those who think humanity is beyond redemption square off against those seeking to defend our planet from the coming invasion, and the fate of two different species lies in the balance.

Liu infuses his world with impressively accurate scientific detail, deftly straddling the boundary between real-world research and imaginative speculation. While Wang’s ultrathin, high-strength nanomaterial – code-named “Flying Blade” because a mere hair’s breadth filament could slice a car in half – is fictional, the molecular self-assembly technique he is trying to develop to scale up production of Flying Blade will be recognizable to any materials researcher. The discussions of invariance and descriptions of modern astronomical instruments and accelerator technology are spot on. Even the Three-Body gaming platform, with its panoramic viewing helmet and haptic feedback V-suit, is firmly rooted in today’s cutting-edge gaming technology.

Unfortunately, after taking so much care to set his plot in motion, in the end, Liu brings his various narrative threads together in uncharacteristically clumsy and ham-fisted fashion. The final pages seem rushed, with long stretches of dry exposition, some of which doesn’t make much sense. Also, the notion that theoretical physicists would be killing themselves in despair over discovering that invariance does not hold across space and time strains credibility. In reality, theorists are made of sterner stuff, like the more pragmatically minded Wang. Most would be thrilled at the prospect of exciting new physics, particularly since whoever solved the conundrum would be a lock for a Nobel prize.

Liu shines most in his gut-wrenchingly evocative depiction of the Cultural Revolution, particularly its bewildering hostility to science. It may strike modern readers as strange that relativity was once considered hopelessly bourgeois, part of a “reactionary academic authority” – especially since Albert Einstein visited Shanghai in 1922. This was an era when teaching general relativity, the Big Bang or the Copenhagen interpretation of quantum mechanics would lead to arrest and torture – and in the case of Ye Wenjie’s father, death. As one of his former students observes when explaining his decision to focus on applied physics, “It’s easy to make ideological mistakes in theory.”

Blind ideology has no place in science. Granted, physicists continue to debate the viability of the Copenhagen interpretation of quantum mechanics, although they need not risk their lives to do so. Yet while the specific points of controversy might be different, a similar strain of denialism still runs through our society. The Three-Body Problem is set in China, but its central theme about the danger of valuing politics and personal belief over objective science transcends time, culture and geography. It’s a warning we should all heed. We may need that science one day to fight off invading alien hordes.

• 2014 Tor Books £17.99/$25.99hb 336pp