Cinema becomes immersive: AI and augmented reality open up new dimensions for production and audiences

In an increasingly digital and connected world, XR technologies and artificial intelligence applied to cinema present themselves as the most promising and revolutionary thrusts towards the new frontiers of creative experimentation confronting the future of the film industry where directors, producers and viewers are invited to explore new languages and test unique experiences in their films.

The cinema continues the constant evolution of its means and canons to bring audiences closer to the stories it tells in new and engaging ways. For this reason, it is the medium that perhaps most easily looks to the future.

In a world where perception increasingly passes through the two-dimensionality of smartphone and computer screens, the new trend of the future is to make vision more and more three-dimensional, while technology is increasingly an indispensable element in enhancing scenes, adding special effects and optimising the overall visual experience of the film.

And this is why industry showcases such as the major international festivals of Cannes and Venice devote entire sections to products made with new technologies as they aim to bring audiences closer to the new dimension of cinema, in which not only sight, but also touch and smell are involved to make the experience increasingly immersive.

Cinema XR: The Future is Now

XR technologies, i.e. augmented, virtual and mixed reality combined in the acronym Experience Realities or Extended Realities, are already used in many sectors, including retail, manufacturing and healthcare. Largely due to the strong push towards digitisation given by the Pandemic, companies have opened up to investing in these technologies convinced by the possible competitive advantages they offer. And cinema, a pioneer of social and economic trends but also a tool for connection, empathy and growth, is looking at these new media as a way forward without hesitation.

This is because viewers are no longer content to look at the images on the screen. They want to be involved in the stories presented to them. Augmented reality responds to this need for participation by offering to enhance the normal perception of reality through digitally generated elements that the audience can see through their personal devices, such as smartphones, or lightweight viewers. Here, the user’s experience takes place in a ‘filter’ or ‘lens’ mode, thanks to which they see additional information of various kinds, i.e. data, videos, photos, links, all within their field of vision.

Virtual reality, on the other hand, transports us into other dimensions that entirely replace the physical one we are in thanks to dedicated devices, i.e. visors that are becoming increasingly sophisticated and offer us absolutely virtual visual and auditory perceptions. With this technology, we are thus taken into a different space and – if we wish – time, where we can perform other activities and interactions than in the real context.

However, these two technologies are not incompatible or alternatives, there is in fact also the possibility of approaching them in a mixed mode, i.e. to have an experience where one uses the visors to access digital content in physical space but adds interactivity with manipulable virtual objects and information.

In cinema, directors and producers are using XR technologies to enrich scenes with advanced visual effects during post-production and to integrate digital elements into physical sets during the production phase. Examples that have changed history include the immersive and visually stunning worlds of Avatar and Jurassic World, which paved the way for this new way of conceiving film production but also satisfy an increasingly demanding audience.

AI at the cinema: Beyond the screen

Artificial intelligence is considered the emerging technology par excellence in all fields, so even cinema, which already features it in so many epic sci-fi films, is confronted with the important opportunities it can provide in terms of cost savings and increased quality.

Its application covers virtually endless possibilities, as demonstrated by films such as ‘Avengers: Endgame’, where it not only assisted animators in creating more realistic and natural movements for digital characters but was also used to create complex visual effects, such as de-aging actors to portray younger versions of characters.  It also played an important role in production optimisation, for example supporting the planning of the shoot, analysing scripts and suggesting the best shots and camera movements.

But it is not a scenario where only the big production companies are moving, especially in Italy we also find a lot of interest in the world of start-ups that propose software to analyse the facial expressions of viewers in real time to adapt the content of the film, improving the viewing experience, but also applications that use artificial intelligence to create scripts based on the preferences of the audience.

Venice and Cannes. What the International Film Exhibitions do

New ways of perceiving reality are at the heart of what is new in the industry, and major international events such as the Cannes and Venice exhibitions have, for some years now, been giving space to the most advanced technologies in sections specially created for them.

Venice was the first international venue to host a competition of XR projects, inaugurating Venice Immersive in 2017. In 2024, it proposed this part of the exhibition dedicated to Extended Reality within the walls of the Isola del Lazzaretto Vecchio, opposite the Riva di Corinto, and presented 63 titles from 25 countries. These included installations, experiences in 6DOF – i.e. experiences in which one can move freely within virtual rooms – Video 360 and VRChat. The latter is a well-known online virtual world platform that allows one to explore a wide variety of worlds in 3D and interact directly with avatars created by other users.

The exhibition therefore allowed its audience to participate in a great collective experience by opening up possible and impossible worlds, the result of the vision of the artists involved. As demonstrated by the film ‘Apartment: In the Realm of Ripley’, an interactive thriller based on artificial intelligence and designed to allow the audience to enter a virtual world, find the culprit and choose the truth. Then, while watching, the audience participates in the plot by making choices that influence the outcome of the story. However, this year’s winner was ‘Impulse: Playing with Reality’, a mixed reality installation created by May Abdalla and Barry Gene Murphy and narrated by Tilda Swinton, a 40-minute interactive documentary that presents the stories of four people facing the decisions that shaped their lives.

And if Italy is always in the vanguard, Cannes 2024 also decided to institutionalise new technologies and, for the first time, opened a section entitled Cannes Immersive and an ad hoc space at the Cineum. This section was presented as the most visionary, innovative and imaginative of the Festival and as evoking exciting new worlds. Also during this edition, Italy participated with ‘Cabiria Experience’, a VR project devised by Rai Cinema together with the Museo Nazionale del Cinema di Torino aimed at helping new generations discover what is considered to be the first kolossal of Italian silent cinema while restoring renewed energy to Giovanni Pastrone’s film, dated 1914. The project involved the use of a 3D graphics engine developed for video games such as Fortnite, and a combination of artificial intelligence tools.

Perspectives and limits beyond the screen

XR and artificial intelligence technologies show the potential to radically transform the enjoyment of film content by offering professionals and audiences a new way to engage and be engaged in immersive and exciting experiences, but still suffer from major limitations in terms of affordability and operational accessibility and raise important questions about the ethical development of these new frontiers by raising profound questions about authenticity, originality and the role of art in an increasingly digital world.

Although major manufacturers such as Meta and Apple are devoting projects and resources to unlocking the world of visors, these still remain expensive technologies for the home user market. As it is still very expensive to keep the host structures operational, the places where it is easiest to enjoy this new cinema today are the film festivals that can afford to allocate dedicated budgets to be sustained within a limited timeframe.

If we look at future scenarios, those of the cinema will depend on our ability to balance technological innovation with the preservation of the very soul of cinematic storytelling. It will be a fascinating but also complex journey that will lead us to redefine our relationship with stories and with reality itself.