Watson Speech to Text supports language and acoustic customizations, but do you need both?

The Watson Speech to Text service has recently released a feature whereby customers can customize the service so that it works better for their specific domain. These customizations can be in the form of a Customized Language Model and/or an Customized Acoustic Model. Customers sometimes get confused by these two different model types and they wonder which they need or if they need both.

The quick summary is that the language customization model tells Watson how the words spoken in your domain are different from normal English (or whatever other base language you are using). For example you might be transcribing speech where the speakers use a specialised terminology. On the other hand, the acoustic customization model tells Watson that the words spoken in your domain might be spoken quite differently than they were spoken in the corpus initially used to train Watson STT. For example, you might have audio samples where the users are using a strong regional accent.

Depending upon your domain, you may need both types of customization, but lets look at them in more detail first.

Customized Language Models

Customized language models tell Watson what words are likely to occur in your domain. For example, when you specify that you wand to use the en-US language, Watson will have a very large (but fixed) list of possible words that can be spoken in US English. However, in your domain, the users might use a specialised vocabulary.  The purpose of the customized language model is to teach Watson how the language in your domain is different from normal English.

The way you build a customized language model is that you provide one or more corpora which are simple text files containing a single utterance per line. It is important to give complete utterances, because accurate speech to text transcription requires that the service knows not only what words might be seen but also in what context the words are likely to occur.

If you are building a model to be used in transcribing film reviews, your corpus might include words like movie,  release,  star and blockbuster. These words are already in the Watson standard dictionary, but including them in your model tells Watson that these words are more likely to occur in your domain than normal (which increases the chance that they will be recognised).

• You might also include the word umpa lumpas in your corpus since people will be discussing them and you need to tell Watson it is a valid word. Since this word is pronounced like it is written, all you need to do is tell Watson that it is a valid word. 
• However, if you are interested in Irish movies, it is likely that people will speak about an actress named Ailbhe or Caoimhe. These common Irish forenames wouldn't be in the Watson dictionary, but it is not enough to tell Watson that they exist. You also need to tell Watson that Ailbhe is pronounced like Alva and Caoimhe is pronounced like Keeva.

The building of this customized language model is probably a relatively simple job. Nevertheless, this customization will probably bring about a dramatic reduction in word error rate. If your audio contains examples of people saying words not in the standard Watson dictionary, then you will never transcribe these properly without a customized language model. In addition, when your speakers say words in the Watson dictionary, the language customization model will increase the chances of these being properly transcribed.

Many users find that the language customization by itself will meet their needs and there is not necessarily any need to combine it with an acoustic model.

Customized Acoustic Models

Customized acoustic models allow you to tell Watson what words sound like in your domain. For example, if speakers in your  region consistently pronounce the word there as if they are saying dare you might need need to build a customized acoustic model to account for this.

At one level building a customized acoustic model is even easier than building a customized language model. All you need to do is upload between 10 minutes and 50 hours of sample audio which is typical of the type of speech that you will be  trying to transcribe with the model that you are building. And then you train the model.

However, if you read the documentation carefully you will see that they say "you will get especially good results if you train with a language model built from a transcription of the audio".  Transcribing 50 hours of speech is a lot of work and so many people ignore this advice. However, I think the advice should read "you are extremely unlikely to get good results unless you train with a language model built from a transcription of the audio". In my experience, training without a language model containing the transcription can very often produce a model whose word error rate (WER) is significantly worse than having no model at all.

To understand why this is the case, you need to look a little closer at how acoustic model training works. For illustration purposes, assume that the problem you are trying to solve is that the utterance a-b-c is sometimes being erroneously transcribed at x-y-z.

• If you train with a language model, Watson will encounter an ambiguous utterance in your training data which it thinks is 70% likely to be x-y-z and 55% likely to be a-b-c. Since your language model doesn't contain x-y-z it will know that it must be a-b-c and it will make adjustments to the neural network to make it more likely that this utterance will be transcribed as a-b-c in the future. Hence, the model gets better.
• On the other hand, if you train without a language model. Watson will encounter an ambiguous utterance in your training data which it thinks is 70% likely to be x-y-z and 55% likely to be a-b-c. Since it has no other information will assume that it must be x-y-z. However, the confidence score is not very high so it will make adjustments to the neural network to make it even more likely that this utterance will be transcribed as x-y-z in the future. Hence, the model gets worse.

Of course the chance of such an error happening is related to the word error rate. In my experience, users rarely put the effort into building a customized model when the WER is low. Mostly people build customized models when they are seeing very hight WER and hence they often see carelessly built acoustic models making the problem even worse.

Another problem people encounter is building an acoustic model from speech which is not typical of their domain. For example, user might be tempted to get a single actor to read out their entire script and get then to record the samples in a recording studio with a good microphone. When their application goes live they might have to deal with audio recorded on poor phone lines, in a noisy environment by people with different regional accents.

Summary

Language and acoustic customizations serve a different purpose - the first deals with non-standard vocabulary while the other deals with non-standard speech sounds. It is possible that you can build a customized language model very easily and this may be enough for your domain. An acoustic model can improve your WER even further, but you should be careful to ensure you build a good one. In particular you should use transcribed data rather than just collecting random samples.

Watson Asynchronous Speech to Text

Speech to Text (STT) transcription can take a long time. For this reason the Watson Speech to Text service offers an Asynchronous API where the caller doesn't need to wait around while transcription is happening. Instead the person requesting the transcription provides details of a callback server which should be notified when the transcription is complete.

This programming style is not too difficult when you get used to it, but there are a number of concepts to learn and one thing that slows down people is that you need a fully functional callback server before you can see any of the other components in action.

In order to help people get started I decided to write a very basic callback server which interacts with Watson STT service, It can help you understand the interaction between the various components and also can also serve as a starting point for a fully functional callback server.

My callback server is implemented in node.js and because it is small, all of the code is in a single file called app.js. Like all node,js programs, it starts with a list of the dependencies which we will use:

const express = require('express');

const crypto = require('crypto');

const bp = require("body-parser");

var jsonParser = bp.json()

const app = express();

const port = process.env.PORT || 3000;

Next we define a variable called secret_key. For security reasons, you probably don't want any random person to be able to send notifications to your callback server. Therefore the Watson STT asynchronous API allows you to specify a secret key that should be used to sign all requests from the Watson STT service to your callback server. A secret which is published in a blog post is not really a secret so you should edit this variable to some secret value which is unique to your deployment. If you don't want to use this security feature, just set the variable value to null.

// var secret_key = null

var secret_key = 'my_secret_key';

This callback server doesn't do much other than write messages in the log to help you understand the flow of messages to and from your callback server. Therefore the log_request() function does that key task for each request.

// record details of the request in the log (for debugging purposes)

function log_request (request) {

  console.log('verb='+request.method);;

  console.log('url='+ request.originalUrl);

  console.log("Query: "+JSON.stringify(request.query));

  console.log("Body: "+JSON.stringify(request.body));

  console.log("Headers: "+JSON.stringify(request.headers));

}

The only thing about this server which is moderately complex is the way it handles signatures. The following function checks whether or not the request contains a valid signature. If the secret_key variable is set to null then no checking is done. When the signature is not valid, it puts a message in the log responds telling you what the signature should have contained.This behaviour is intended to be helpful for developers debugging interactions, but you would probably want to turn it off for production systems because it would be helpful for hackers.

// check if the signature is valid

function check_signature(request, in_text) {

  // check the request has a signature if we are configured to expect one

  if (secret_key) {

    var this_signature = request.get('x-callback-signature');

    if (!this_signature) {

      console.log("No signature provided despite the fact that this server expects one");

      throw new Error("No signature provided despite the fact that this server expects one");

    } else {

      console.log("Signature: "+this_signature);

    }

    // Calculate what we thing the signature should be to make sure it matches

    var hmac = crypto.createHmac('sha1', secret_key);

    hmac.update(in_text);

    hmac.end();

    var hout = hmac.read();

    var expected_signature = hout.toString('base64');

    console.log("Expected signature: "+expected_signature);

    if (this_signature != expected_signature) {

      err_str = "Actual signature \""+this_signature+"\" does not match what we expected \""+expected_signature+"\"";

      console.log(err_str);

      throw new Error(err_str);

    }

  } 

}

The server needs to handle POST requests coming from the Watson STT server when the status of any transcription service changes. All we do is log the request for debugging purposes. If the signature matches the body of the POST, we give a status of 200 and respond with OK. Obviously a production server would be expected to do something more useful.

// Handle POST requests with STT job status notification

app.post('/results', jsonParser, (request, response) => {

  log_request (request);

  if (!request.body) {

    var err_text = 'Invalid POST request with no body';

    console.log(err_text);

    response.status(400);

    response.status(err_text);

  }

  check_signature(request, JSON.stringify(request.body));

  // for now just record the event in the log

  console.log('Event id:'+request.body.id+' event:'+request.body.event+' user_token:'+request.body.id);

  // The spec is not clear about what we should respond to just say OK

  response.type('text/plain');

  response.send("OK");

})

When registering your callback server, Watson STT issues a GET request with a random challenge_string to see if your server is up an running. If the signature on the request matches the content of the challenge_string then we simply echo back the challenge_string to let the Watson server know we are functioning OK. If the signature is wrong we issue an error response and the registration of the callback server will fail.

// Deal with the initial request checking if this is a valid STT callback URL

app.get('/results', (request, response) => {

  log_request (request);

  if (!request.query.challenge_string) {

    console.log("No challenge_string specified in GET request");

    throw new Error("No challenge_string specified in GET request");

  }

  check_signature(request, request.query.challenge_string);

  response.type('text/plain');

  response.send(request.query.challenge_string);

})

Finally the app starts listening for incoming requests:

app.listen(port, (err) => {

  if (err) {

    return console.log('something bad happened', err);

  }

  console.log(`server is listening on ${port}`);

})

I have an instance of this callback processor running at https://stt-async.eu-gb.mybluemix.net/results but it is not really any use to you since you won't be able to see the console log messages. You can also download the complete sample for code from GitHub and host it either in BlueMix or the hosting platform of your choice,

Adding a speech interface to the Watson Conversation Service

The IBM Watson Conversation Service does a great job of providing an interface that closely resembles a conversation with a real human being. However, with the advent of products like the Amazon Echo, Microsoft Cortana and the Google Home, people increasingly prefer to interact with services by speaking rather than typing. Luckily IBM Watson also has Text to Speech and Speech to Text services. In this post we show how to hook these services together to provide a unified speech interface to Watson's capabilities.

In this blog we will build upon the existing SpeechToSpeech sample which takes text spoken in one language and then leverages Watson's machine translation service to speak it back to you in another language. You can try the application described here on Bluemix or access the code on GitHub to see how you can customise the code and/or deploy on your own server.

This application has only one page and it is quite simple from the user's point of view.

• At the top there is some header text introducing the sample and telling users how to use it. 
• The sample uses some browser audio interfaces that are only available in recent browser versions. If we detect that these features are not present we put up a message telling the user that they need to choose a more modern browser. Hopefully you won't ever see this message.
• In the original sample there are two drop down selection boxes which allow you to specify the source and target language. We removed these drop downs since they are not relevant to our modified use case.
• The next block of the UI gives the user a number of different ways to enter speech samples:
• There is a button   which allows you to start capturing audio directly from the microphone. Whatever you say will be buffered and then passed directly to the transcription service. While capturing audio, the button changes colour to red and the icon changes  - this is a visual indication that recording is in progress. When you are finished talking, click the button again to stop audio capture.
• If are working in a noisy environment or if you don't have a good quality microphone, it might be difficult for you to speak clearly to Watson. To help solve this problem we have provided you with some ample files hosted in the web app. To play one of these samples click on one of the buttons to play the associated file and use it as input.
• If you have your own recording that you can click on the  button and select the file containing the audio input that you want to send to the speech-to-text service.
• Last, but not least, you can drag and drop an audio file onto the page to have it instantly uploaded
• The transcribed text is displayed on an input box (so you can see if Watson is hearing properly) and sent to either the translation service (in the original version) or the conversation service in our updated service. If there is a problem with the way your voice is being transcribed, see this previous article on how to improve it.
• When we get a response from the conversation or translation service we place the received text on an output text box and we also call the text-to-speech service to read out the response and save you the bother of having to read.

I know that you want to understand what is going on under the covers so here is a brief overview:

• The app.js file is the core of the web application. It implements the connections between the front end code that runs in the browser and the various Watson services. This involves establishing 3 back-end REST services. This indirection is needed because you don't want to include your service credentials in the code sent to the browser and because your browser's cross site script protections will prohibit you from making a direct call to the Watson service from your browser. The services are
• /message - this REST service implements the interface to the Watson Conversation service. Every time we have a text utterance transcribed, we do a POST on this URL with a JSON payload like {"context":{...},"input":{"text":"<transcribed_text>"}}. The first time we call the service we specify an empty context {} and in each subsequent call we supply the context object that the server sent back to us the last time. This allows the server to keep track of the state of the conversation.
  Most conversation flows are programmed to give a trite greeting in response to the first message. To avoid spending time on this the client code sends initial blank message when the page loads to get this out of the way.
• /synthesize - this REST service use used to convert the response into audio. All that this service does to convert a get on http://localhosts:3000/synthesize?voice=en-US_MichaelVoice&text=Some%20responsevoice=en-US_MichaelVoice&text=Some%20response into a get on the URL  https://watson-api-explorer.mybluemix.net/text-to-speech/api/v1/synthesize?accept=audio%2Fwav&voice=en-US_MichaelVoice&text=Some%20response this will return a .wav file with the text "some response" being spoken in US English by the voice "Michael". 
• /token - the speech to text transcription is an exception to the normal rule that your browser shouldn't connect directly to the Watson service. For performance reasons we chose to use the websocket interface to the speech to text service. At page load time, the browser will do a GET on this /token REST service and it will respond with a token code that can then be included in the URL used to open the websocket. After this, all sound information captured from the microphone (or read from a sample file) is sent via the websocket directly from the browser to the Watson speech to text service.
• The index.html file is the UI that the user sees. 
• As well as defining the main UI elements which appear on the page, it also  includes main.js which is the client side code that handles all interaction in your browser.
• It also includes the JQuery and Bootstrap modules. But I won't cover these in detail.
• You might want to have a closer look at the client side code which is contained in a file public/js/main.js:
• The first 260 lines of code are concerned with how to capture audio from the client's microphone (if the user allows it - there are tight controls on when/if browser applications are allowed to capture audio). Some of the complexity of this code is due to the different ways that different browsers deal with audio. Hopefully it will become easier in the future. 
• Regardless of what quality audio your computer is capable of tracking, we down sample it to 16bit, mono at 16 Khz because this is what the speech recognition is expecting.
• Next we declare which language model we want to use for speech recognition. We have hardcoded this to a model named "en-GB_BroadbandModel" which is a model tuned to work with high fidelity captures of of speakers of UK English (sadly there is no language model available for Irish English). However, we have left in a few other language models commented out to make it easy for you if you want to change to another language. Consult the Watson documentation for a full list of language models available.
• The handleFileUpload function deals with file uploads. Either file uploads which happen as a result of explicitly clicking on the "Select File" button or upload that happen as a result of a drag-and-drop event.
• The initSocket function manages with the interface to the websicket that we use to communicate to/from the speech_to_text service. It declares that the showResult function should be called when a response is received. Since it is not always clear when a spaker is finnished talking, the text-to-speech can return several times. As a result the msg.results[0].final variable is used to deremine if the current transcription is final. If it is an intermediate result, we just update the resultsText field with what we heard. If it is the final result, the msg.results[0].alternatives[0].transcript variable is also used as the most likely transcription of what the user said and it is passed on to the converse function.
• The converse function handles sending the detected text to the Watson Conversation Service (WCS) via the /message REST interface which was descibed above. When the service gives a response to the question, we pass it to the text-to-speech service via the TTS function and we write it on the response textarea so it can be read as well as listened to.
• In addition there are many other files which control the look and feel of the web page, but won't be described in detail here e.g. 
• Style sheets in the /public/css directory
• Audio sample files in the /public/audio directory
•  Images in the public/images directory
• etc.

Anyone with a knowledge of how web applications work, should be able to figure out how it works. If you have any trouble, post your question as a comment on this blog.

At the time of writing, there is an instance of this application running at https://speak-to-watson-app.au-syd.mybluemix.net/ so you can see it running even if you are having trouble with your local deployment. However, I can't guarantee that this instance will stay running due to limits on mypersonal Bluemix account.

Watson Speech to Text with Node.js

Talking to a computer makes you feel like your living in the future. In this post I will show how using node.js to expand on the Watson Speech to Text (STT) example to improve the accuracy of the transcription.
As a base I will take the cognitive car demo and try write a STT system for that. Once Speech to text can correctly interpret what a person asks the car we will want to send those commands to The Watson Conversation Service Car demo. But I will deal with connecting STT to WCS in another post.
Log into bluemix and set up a STT service. Create new credentials and save them as you will use them in all the following programs. This username and password are used everywhere below.
Next assuming you already have node installed you also need the watson developer cloud SDK
npm install watson-developer-cloud --save
and the speech-to-text-utils
npm install watson-speech-to-text-utils -g

Get an audio file of the sorts of things you would say to this Car demo. I asked questions like the ones below

turn on the lights
where is the nearest restaurant
wipe the windscreen
When is the next petrol station
When is the next gas station
does the next gas station have rest rooms
How far are the next rest rooms
please play pop music
play rock music
play country music


I have made an audio file of that (I had a cold). You can get it here or record your own version.
In speech_to_text.v1.js in the sdk examples file put in the username and password you got in 1. above. And point at the sound file of commands from 3.

const speech_to_text = new SpeechToTextV1({
  username: 'INSERT YOUR USERNAME FOR THE SERVICE HERE',
  password: 'INSERT YOUR PASSWORD FOR THE SERVICE HERE'
});

fs.createReadStream(__dirname + '/DavidCarDemo.wav').pipe(recognizeStream);

Now look at transcription.txt. We want to improve the accuracy of this transcription. Now create a model that we will train so that the STT understands our speech better.

Running node createmodel.js on this code gives you a custom_id needed from here on.

{ "customization_id": "06da5480-915c-11e7-bed0-ef2634fd8461" }

5. If we tell STT to take in the car demo corpus. It learns from this the sort of words to expect the user to say to the system
watson-speech-to-text-utils set-credentials
and give it the username and password same as above.
watson-speech-to-text-utils customization-add-corpus
Will ask you for a conversation workspace. Give it car_workspace.json downloaded from the car demo. Adding a conversation workspace as a corpus improves accuracy of the words that are unusually common in our conversation.
Now we want to improve accuracy on words that are not common in the corpus. For example "Does the next" is heard as 'dostinex" currently.
watson-speech-to-text-utils corpus-add-words -i 06da5480-915c-11e7-bed0-ef2634fd8461
and give words.json which looks like

{
  "words": [{
    "display_as": "could",
    "sounds_like": [ "could" ],
    "word": "culd"
  }, {
    "display_as": "closeby",
    "sounds_like": [ "closeby" ],
    "word": "closeby"
  }, {
    "display_as": "does the next",
    "sounds_like": [ "does the next", "dostinex" ],
    "word": "does_the_next"
  }
]
}

The speech utils allow you to see what words you have in your model. Words added to not overwrite old ones so sometimes you need to delete old values. watson-speech-to-text-utils customization-list-words -i 06da5480-915c-11e7-bed0-ef2634fd8461
watson-speech-to-text-utils corpus-delete-word -i 06da5480-915c-11e7-bed0-ef2634fd8461 -w "word to remove"

Finally in transcribe.js you need to tell it to use the model we have just made

const params = {
  content_type: 'audio/wav',
  customization_id: '06da5480-915c-11e7-bed0-ef2634fd8461'
};

The code I have used is up on github here.
Now we get a more accurate transcription of the voice commands. By training on the Conversation corpus. Then fixing other common errors using a words.json file.