Berlin team creates multi-sensory, immersive gesamtkunstwerk with technological help

Why don't we let what we hear merge with what we see? It all started with this question from Michaela Catranis. Together with data scientist Nikolay Jetchev, the concert pianist and composer of contemporary classical music "originally had the idea of translating live music and sound AI, supported in real time, into images in order to create a new kind of multisensory Gesamtkunstwerk," says Timo Hagenow, co-founder and CEO of the Berlin-based AI innovation lab LF1. Together with his colleagues and AI experts Duncan Blythe and Alexander Schlegel, he turned the idea into reality.

Clusters of stars in the black nothingness, colored lights that suddenly flash through the clouds and spread out - these are recordings from the "Hubble" space telescope that dance to the sounds of Michaela Catranis' piece "Synthingout," thus bringing the auditory to life visually as well. This translation of sound into images happens with the help of modern AI technologies using various neural networks. How this is done is described on the website of SHEEN AI - the name of the project: According to the sound, visual AI "paints" visuals on a dynamic canvas using its imagination. The imagination is based on visual concepts derived from visual reference inputs such as images and videos and brings them to life - even live on stage. Subsequently, audio AI perceives sound and breaks it down into auditory components, which in turn must be combined with visual features to create the desired look and feel of the image.

Immersion in water drops and acrylic paints

Synthingout is not the only piece of music in which this innovative technology has already been used. According to Hagenow, "diverse but little-known" sound works have already been visualized under SHEEN AI's brand. The AI does not always "paint" with images from outer space; water drops or an acrylic painting by artist Pawel Czerwinski - conducted by the technology - also allow all the senses to be immersed in the art experience.

SHEEN AI, which was launched in January 2022 as a project of LF1, is only at the beginning:

"We are currently working on a number of exciting collaborations, especially with electronic music artists," explains Hagenow, who says he is very enthusiastic about music and visual art. The entrepreneur is not yet allowed to reveal more details, but the vision of him and his small team is clear: "Our focus is on exploring the artistic potential of a symbiosis between artificial intelligence and traditional art forms," says the businessman, who is currently building up a network of freelancers and artists such as musicians, visual artists and VJs. "Our goal is to use AI to create a new art medium in which the auditory merges with the visual to create a multisensory, fully immersive environment - as a total work of art."

Target group: From the artist to the brand

This total work of art could be interesting for visual artists who want to bring their paintings to life, for example, but also for musicians and labels for online use or for venues such as concert halls that want to show audio-reactive visuals.

"Event organizers or operators who want to show or use our visuals as a gimmick or for advertising purposes are also potential target groups," Hagenow says. Another way to bring this immersive toal work of art to life is through concerts and live performances - a first prototype of SHEEN AI could already be tested at the CAA Award and the Cres.Biennale 2020: Human Machine. "Such references naturally make it easier for us to find new projects and help us to be heard," says the Hamburg native happily.

Berlin best platform for visual digital art

The live concerts planned for the first half of 2023 should also literally strike a chord. "Especially in the field of classical music - hopefully with a large orchestra," Timo Hagenow sees the potential. Live DJ sets are also planned, he says, as well as the first larger installations. There's no doubt in his mind that SHEEN AI is in exactly the right place in Berlin, since "with its music and club culture, it simply offers by far the best platform for visual digital art in Germany, if not Europe or the world."

With its active startup scene and its attractiveness for - international - experts in the field of AI, the German capital is also the right choice for SHEEN AI's second intention: "At our core, we see ourselves as technology developers with a corresponding need for resources," says Hagenow, "according to the motto: The more people work and have an effect, the better. In this respect, we also have the ambition to grow." Thus, the entrepreneur would soon like to work not only exclusively project-based, as is the case at the moment. "In the future, we also want to license the software," says Timo Hagenow. SHEEN AI thus hopes for more and more immersive works of art in which what is heard merges with what is seen.