Refined Inception 1080p Mkv Subtitles Extract
Ayana Munsen
Let me be very clear: I am not writing this article to advertise myself, that maybe, eventually, will come later... Every time I make a product I end up writing somewhere, on some website, an article that tells the genesis of that product, its story.
So I am not here to sell you the product whose story I am about to tell you, but to chat, and if downstream of this reading you would like to interact with me, ask me some questions, ask me for insights, even give me some criticism, more or less constructive, I am here, absolutely happy to debate and learn something new.
Again, before we begin two words about yours truly: my name is Alex, I live and work in Milan, Italy, I studied computer engineering and then, "fortunes" in life led me to become first an editor and then a video producer.
Now my official label, as far as my occupation is concerned, is "director of TV commercials and promotional videos," however, I deal with all aspects of production and especially post-production in a small company called Exmachina, also in Milan.
Throughout my career as a video maker, however, I have always made small tools, tiny programs, utilities that we use internally in the company as an aid to the management of the video and audio production processes (I probably forgot to tell you that I have also been doing podcasts for 12 years, my last show, if you know some Italian, is Techno Pillz in which I talk about my work as a programmer and video maker).
After the great crisis of 2008, which was overwhelmingly felt in Italy in 2011, I had to kind of find a way to reinvent myself, so I started in 2017 to refine some of the applications I had made and then sell them directly on the Ulti.Media website.
Initially it was not even a second job, as much as, let's say, a kind of hobby, but for a few years now in this second job has become a very important part of both my personal and professional life, with a market that is starting to get interesting.
Towards the end of 2022, and in January 2023, I started working on a very ambitious application, a voice synthesis application, with the goal of becoming an interesting replacement for the "cheaper" versions of video tutorials: the ability to have not simply a text-to-voice generator, but a real project management system as if you were directing a virtual speaker.
I don't know if this program will be able to see the light of day before the end of 2023, I'm still working on it and it's quite a long process, at any rate, also earlier this year, I had started to do some experiments with Whisper, for transcribing audio, something that is quite handy for me to find keywords in podcasts (and which I've already done using Siri in another application of mine, FCP Video Tag).
In my head, during my car rides between home and office, I began to form the idea of an application that would not only pull out subtitles but also allow text editing, something derived from the same text editing engine as the other application I am developing, the one that does the exact opposite, from text to voice.
So I come across Macwhisper, an excellent application by Jordi Bruin, download it, try it out, and decide not to pursue developing my own application: this competitor has already done it all and done it very well, too!
(Note that in an article in which I talk about one of my products, I quote one of my competitors and give him high compliments: at the very least in marketing school they would have flunked me with a grade worse than unclassifiable...).
Sometimes I think Christopher Nolan's movie Inception is based on the ideas that become mental woodworms in the space between my ears... I kept thinking about how I would develop the application because while Macwhisper was perfect in some ways, it was more designed for transcribing audio, and not for managing subtitle streams with easy, visual, in-sync editing...
This is probably referred to in psychology as "megalomania." I think it's more unconsciousness, however, often and often the most fun things I've done in my life I've done just with a considerable amount of unconsciousness.
Well, yes, I decide to develop my application anyway and focus it on a different, probably fuller workflow than Macwhisper's, and I start testing with Whisper's Large model, made available in open source by the people who created it: OpenAI, yes, the same people as ChatGPT.
Let's make a nice technical digression, since we're here among friends, pretend we're in a pub, and we've made it to the third round of beer, only instead of talking about soccer or some other sport, we're nerdy enough to get excited about something technological...
So why is Whisper so important? And what ties it to ChatGPT? I will try to explain things in a simple and certainly inaccurate way, so those who understand a little about artificial intelligence, plug their ears...
Whisper is a transformer, and while Siri, Alexia, and Google try to understand the word or phrase that was spoken, Whisper goes a little bit wide and also tries to understand the context of each phoneme, and through a very broad knowledge base as well as statistical calculations, tries to make each word follow the word that makes the most sense, consistent with what is being spoken.
Whisper's wide model takes up a good 3GB, a little more in memory when it runs, but it is still to be considered a "basic" model, probably ChatGPT four is at least 100 times bigger - which is why it will be difficult to run something like this on our computers.
Rose-water technical details aside, this is one explanation of why Whisper is so much more accurate and also succeeds where other transcription systems stall: in the unfortunate event that it does not understand a word or part of a word, it "reconstructs" it from what comes before and what comes after.
Another major struggle was to "teach" (although perhaps the correct term would be to specify) Whisper to also generate punctuation marks, such as periods, commas, question marks, and escalation marks.
One of the great peculiarities of Whisper is that it can also figure out what punctuation marks are, but this is not something it does automatically, you have to specify some input parameters so that it understands that the context has to include them as well.
In short, teased by the Twitter discussion and the endorsement, I started playing around with this system, trying to figure out what I could get out of it and what the limitations of the technology were.
I think I managed to make a very first prototype, with, of course, a horrible user interface within 7-10 days. And, of course, I started testing it with all the work I was doing in March 2023, projects of various kinds: from pharmaceutical video presentations to a video game trailer to some tutorials.
Here I encountered the first limitations of Whisper, which performs exceptionally well when the audio is quite clear, content with a speaker or interviews, while it goes deeply into crisis in case music or sound effects are present.
Initially, I told myself that this would not be a big problem, just specify in the requirements that the application was best suited for spoken content, nevertheless I felt the need to refine it, so I rolled up my sleeves and wondered if there was a possibility of parsing the audio in such a way as to find the points where there was speech and then eliminate those where there was only music or sound effects.
I succeeded by taking advantage of some libraries provided by Apple for analyzing sounds.Subsequently, one thing I have been working on in parallel with both this app and the one that does text-to-speech is the highlighting of sentences in sync with the video (or, rather, with the audio).
I always thought that when doing a speech review, physically, the people in charge have in front of them a sheet, printed in Courier font, with all the features of the script.For this very reason I wanted the words and sentences to be written in this same style, not so much for skeuomorphism, but because it is still the most effective and efficient method of reading the text while listening to the related audio.
When there is information that varies over time, the concept of a timeline is one of the strongest: it transforms the horizontal axis into that of time, so the playhead moves from left to right as the playback continues.
Showing in a two-dimensional screen information that evolves over time is never a trivial matter, not least because it involves making user interfaces and real-time interaction methods, which in some ways have more to do with video games than with video writing programs...
This is why I am particularly proud of this part of the code; not only is the segment highlighted with the sentence being spoken, but it is also easy to interact with it by correcting any transcription errors.
But then I thought about all the times there would be a need for an upgrade, and downloading 3 GB of stuff, most of it always the same, would be particularly troublesome, especially in areas of the world where connection speeds are not comparable to those of the big European or North American metropolises.
So I set to work on making a system that would allow the downloading of the template only the first time, and among other things in "shared" mode, so that in case I wanted to develop another application based on the same technology, it could safely use the model already downloaded (spoiler: thing I have actually already done with FCP Video Tag).
At the end of April I had to go to Florence to attend a medical conference and do several interviews with many doctors, in a total of 12 languages, including English, French, Spanish, Italian, German, Polish, and Brazilian Portuguese.
"Of course yes!", I replied, and the fact that I had a real use case, with tight deadlines and a not inconsiderable qualitative demand, led me to work heavily on the application, so as to improve as much as possible the interface and also the possibility of exporting the subtitles in different formats, including text with total transcription, unsynchronized, and later also RTF with time code indication in a different style.
7fc3f7cf58