Interactive learning application with basic web APIs
I was looking at the possibility of creating an application that can aid both children and adults to learn a new communication language from their home during the time of lockdown. My requirements were very clear.
1) It should be a web application so that users with basic android phone can use it with out an app installation.
2) It should be light-weight so that users with slow internet connection can access it.
3) It should be designed with free tools available in the market.
4) Should be interactive.
Dr. Abjith Debnath helped me to understand the steps that users should follow to successfully learn a language.
Light Weight and Easy to Use
In-order to make the app light weight and accessible, I have used HTML, CSS and JS. I have decided that I will not use any frameworks at-least for the prototype. This decision has helped me to understand the real power of CSS Grid and Flexbox.
In order to make the instructions more clear I have also added a voice based instruction. I have made the prototype only for Hindi language.
Listening
The more you listen, the more you will become better in using the language. That is the reason why we all are very good at using out mother tongue. So for listening part I have designed two modules.
Module 1: Listen & Recognize
This module involves listening to two voices and determine whether the voices are matching or not. This was an easy task to implement with basic DOM manipulation.
Module 2: Listen & Speak
This module involves listening to an audio and record your voice while trying to replicate the voice you have heard. The user can then compare the voices themselves and rerecord it until he/she feels satisfied.
Implementation
For recording the voice I have used MediaStream Recording API and existing example from Mozilla documentation.
The MediaStream Recording API, sometimes referred to as the Media Recording API or the MediaRecorder API, is closely affiliated with the Media Capture and Streams API and the WebRTC API. The MediaStream Recording API makes it possible to capture the data generated by a MediaStream or HTMLMediaElement object for analysis, processing, or saving to disk. It’s also surprisingly easy to work with.
Speaking
After the user is comfortable with listening skills he can move on to improve his speaking skills. I have designed two modules for this purpose.
Module 3: Listen & Read
This module involves two steps:
1) The user will listen to the prerecorded audio.
2) The user records the audio. The audio is converted into text and compared with the original text.
This will help the user to improve pronunciation.
Implementation
I have used Web Speech API for the implementation. Downside is that it is not supported by Mozilla Firefox.
We shall initiate the api with following code.
recognition.start() was used to start the recording. Once user stops speaking the recording stops automatically.
Module 4: Read & Speak
This will be a self-help module where the user read a sentence, record the same in his own voice and analyse his voice by comparing the voices.
Present State
I have only prepared a test version of each module with one example each.
This is accessible at the following git repository.
The details can be found here
https://github.com/Vidhyanandcs/learn-language
Future Plans
Complete Module 5 (writing) and Module 6 (Listen and write)
Module 5 involves watching a GIF video of a virtual hand movement while writing a letter or a word. The user will then use his finger to replicate the stroke on a virtual book on the screen. The movement of his hand is analyzed and given a score.
Module 6 involves listening to an audio, writing the letter or word on a piece of paper and uploading the same to the system. The text will be extracted and compared with original text to give a result.
If we can work on these modules and create the application for different languages, people across the world who are not well off can use these resources to learn during this difficult time of covid-19.