In a manner of speaking | 2022
Video Installation, AI Lip-sync model
Since the collapse of the Soviet Union, the Russian language has remained Kazakhstan's second official language, but the most used and dominant. "In a manner of speaking" is an irony of the current post-colonial linguistic situation in Kazakhstan. The people on the recordings who are perfectly “speaking” Kazakh are Kazakhs, who do not speak their supposed mother tongue at all or speak at a very basic level. The videos have been manipulated using lip-sync model to simulate a speech, a speech in their forgotten mother tongue.
For this project I used a pre-trained neural network called “Wav2Lip” (Deepfake) in order to produce accurate lip movements on a static image of specific people. The inputs were a text-to-speech voice reading a Kazakh poem and static images as mp4 of people. The output was quite disheartening. In most of the output videos their mouth was out-of-sync in general. I didn't understand what the problem was for a long time, as there were so many accurate examples that were created using this particular model. Until a colleague of mine, Behiye Erdemir noticed that the problem could be exactly the lack of a Kazakh language dataset in this neural network.
In particular, there are problems of patriarchal and post-colonial perspectives in science and Data Science. The dataset used in the aforementioned “Wav2Lip” neural network is a collection of thousands of spoken sentences from BBC television.
Perhaps it’s no surprise to learn that this data-driven organization works best when more data is available to it. As an example, Google has a small amount of content in Kazakh language. And since non-Western languages like Kazakh has lack of data and with only nine million speakers, Kazakhstan isn’t really a big enough market for Google to invest in solving this problem.
Can we think of an effective counterculture in this case, if instead of being in confrontation with the dominant culture, we build an alternative way such as training neural networks that supports non-Western languages?