London Mayor AI Deepfake Nearly Resulted In "Serious Disorder"

AI is now good enough to create convincing deepfakes, and there is no easy way to prevent misinformation being disseminated.

How will AI be used to influence elections and undermine reputations?

Sadiq Khan, London's Mayor, has said that a deepfaked recording of him apparently making controversial statements before Armistice Day almost led to serious disorder.

Armistice Day is marked annually in the UK as the moment at which World War One ended on 11 November 1918. Memorial ceremonies are held across the country on the nearest Sunday to that date, known as Remembrance Sunday.

Audio Deepfakes: The New Frontier In Election Disinformation
Audio deepfakes are becoming a key tool in election disinformation, with AI technologies enabling realistic voice clones.

Stoking Tensions

The audio clip, which was disseminated widely on Twitter and other social media platforms, ostensibly featured Khan saying that Pro-Palestinian marches should take precedence over the Remembrance weekend ceremonies, which were planned for the same day. The AI-generated voice described Remembrance weekend in derogatory terms, using a swear word.

The clip was created to sound like a candid recording of a private meeting. "What's important and paramount is the one-million-man Palestinian march takes place on Saturday," the voice says. Khan's faked voice also says that he controls the Metropolitan Police, and that the Prime Minister's meeting with the Met Chief was a "waste of time" for that reason. "The British public need to get a grip", the voice adds.

Red Rag To A Bull

The recording was clearly designed to anger a wide cross-section of British people, since it belittled the sacrifices of those who died in the two World Wars. "The timing couldn't have been better if you're seeking to sow disharmony and cause problems," Khan told the BBC.

What was being said was a red rag to a bull for the far right and others. But what concerned me the most was if you're an innocent listener of this. Because it's a secret undercover recording, in inverted commas, because it sounds like me, because of the timing and the context.

The faked audio has been traced to a pseudonymous TikTok account. The Metropolitan Police have examined it, but decided that no criminal offence has been committed. There is, in fact, no law that currently covers this kind of activity, even though the consequences could have been severe.

It is likely that, as key elections draw near in the UK, US, and elsewhere around the world, more of this kind of footage will be created with the intention of undermining reputations and sowing discord. In some cases, violence could result, as it almost did in this instance. As yet, there is very little means of avoiding some of the worse scenarios that AI raises.

Subscribe to our newsletter and follow us on X/Twitter.

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to REX Wire.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.