Audio-Text Alignment

Audio-Text Alignment / Forced Alignment

Necessity and Application

Sometimes we have the text of an audio speech or a specific video, but it is not clear which part of the text corresponds to which part of the audio. This information is essential for:
- Simultaneous display of text and audio/video to the user
- Displaying text as subtitles on video and audio

The Audio-Text Alignment program is designed for this purpose.

Main Challenge

This program uses an external speech recognition service. Currently, speech-to-text services for lectures and meetings do not have good accuracy. Therefore, the process has become more complex.

Process Steps

Step 1: Format Conversion

First, we convert the video or audio to a format usable by the speech-to-text module. For this, the audio quality must be good.

Step 2: Segmentation into Smaller Chunks

Then, by detecting silences within the speech, we divide the audio into smaller chunks and send them to the speech recognition module.

Step 3: Multiple Segmentation (if needed)

For higher accuracy, in parts where there is no long silence, multiple different segmentations can be used.

Step 4: Speech Recognition

The output of the speech recognition module is one or more recognized sentences that roughly correspond to the original audio chunk.

Step 5: Dynamic Alignment

After normalizing the original speech text and the recognized audio chunk texts, we use a dynamic algorithm to align audio timestamps with potential positions between words in the original text. This algorithm is based on calculating the Levenshtein distance between the concatenated recognized sentences and the original text.

Output

The output consists of segmented parts of the original text, with the timing of each text segment recorded alongside it. This output is used to generate standard subtitle files that can be played in media players.

Successful Results and Applications

The output has been successfully used for:
- Text and audio of speeches on the Leader.ir website
- Alignment of Quran recitation (Tarteel) with Quran text

Applications

• Automatic Subtitling: Generating subtitle files for videos and speeches
• Simultaneous Text and Audio Playback: Displaying text synchronized with audio playback
• Speech Text Mining: Enabling text-based search within audio content
• Educational Resources: Creating alignment between lesson text and audio files
• Quran Recitation: Precise alignment of verses with recitation

Benefits

• Good Accuracy Despite Challenges: Despite the limited accuracy of speech recognition services, the dynamic alignment algorithm increases accuracy
• Standard Output: Produces subtitle files playable on standard media players
• Successful Real-World Applications: Verified on speeches from Leader.ir and Quran Tarteel alignment
• Silence Management: Intelligent use of silence detection for optimal segmentation
• Dynamic Algorithm: Uses Levenshtein distance to find the best alignment