Matching patterns and getting their values in Watson Assistant/Conversation

When IBM Watson Assistant (formerly known as Watson Conversation) is deciding how to respond to a user's utterance it is vital that it correctly identifies the intent (what the user wants to do) and the entities (what are the things involved in the intent). For example, if the user says "I want to buy a book" - the intent would be #MakePurchase and the entity @ItemOfInterest would have a value of "book".

In earlier releases of Watson Assistant, the only way to specify possible entity values was either by manually specifying a list of possible values or else by selecting one of the predefined system entities such as @sys-date. Sometimes this works quite well, but other times (e.g. when you are expecting an email address or an account number) is not feasible to list all of the possible values that people might enter.

Luckily, the latest version of the Watson Assistant service allows you to specify allowable entity values with a regular expression. Unfortunately, people sometimes find it hard to retrieve the matched value from a pattern match. If you are not careful you will be told that an email address was specified and not what exact email address was given. Therefore this blog post works through a very simple conversational design to explain what you need to do.

First off, you define an intent. We will call our intent #sendMessage and we give Watson a few examples of what the user might say when they want to send a message.

Then we create a @contact_info entity which we expect users to specify when they are sending a message. To complete this entity, the user types a message indicating that they want to send a message. We expect that the message will also contain details of where to send the message, either an email address or a phone number (the phone number can be specified in US style or in the e164 standard common in other parts of the world).

This picture shows how the entity definition will look. Don't worry if you can't read the regular expressions in the screenshots, you can download the workspace design.

Now you need to insert a dialog node to handle requests to send messages. We create a node in our dialog flow which is triggered when Watson detects that the user's intention is to send a message. We know that it is necessary to have contact information to send a message, so if the user didn't supply this we will prompt them.

Then we need sub-nodes which deal with sending either emails or phone messages. We select which to activate depending the value of the @contact_info variable, which will be either email, us_phone_num or e164_phone_num.

When sending a message, it is not enough to know that the user gave us an email address - we need to know the exact email address given.  To do that, we  need to define a variable whose value will be specified as  "<? @contact_info.literal ?>".  The screen shot below shows the dialog node for sending a phone message.

This is the end of our very simple BOT. If you want to see this in action, download the design file here and  import it into your own Watson Assistant instance. Here is a screenshot of what I saw when I clicked on the "Try it out" button to see the bot in action.

In summary, regular expression entities can be really useful, so long as you remember to use the @entity_name.literal syntax to get the actual content that was matched rather than simply which rule was fire.

Naming Intents

How should you name intents? Heres one way and an explanation as to why.

In this post we described clustering a topic into intents. The naming scheme I used was TopicIntent.

When you go to improve accuracy you will merge and split intents. You tend not to do this outside Topics. I find that if you have the topic name in the intent when you do these changes it is easier to keep your brain in one context.

Cluster Topics

"Happy families are all alike; every unhappy family is unhappy in its own way." the Anna Karenina principle

Some Topics cover loads but you don't really care about the individual intents inside. For example if you have a Complaints topic that could cover all sorts of things people moan about.

No one wants a message back saying "This robot cares we have lost your bags". A complaint question will have to be passed onto a person. If we can tell that person that we have a complaint they can then decide what to do next. If you do not break down complaint topic into intents though all sorts of questions will be in one intent. It will deal with damage, delays, queues, lost items, dirty conditions etc. This giant varied intent will suck in other questions damaging your overall system accuracy.

With a varied topic like complaints that your chatbot cannot handle by itself. If you make one giant intent it will damage your overall accuracy. But because complaints tend to be about a few things at once, 'The food was terrible and the portions were small', there is often not one solid intent anyway. By labelling all complaints ComplaintIntent it is possible to ignore the intent part as getting the topic right is good enough.

In our accuracy tests we can strip the intent part off and say that if we land in Complaint that is good enough. But not create on giant intent that covers too much and that will suck in all other questions.

This issue of big topic particularly happens with Off Topic topics where questions are out of scope, silly or just cover large areas that you can't really answer.

There are other ways to label intents. This TopicIntent method is what I use. If you have a different way please mention it in the comments.

Adding a speech interface to the Watson Conversation Service

The IBM Watson Conversation Service does a great job of providing an interface that closely resembles a conversation with a real human being. However, with the advent of products like the Amazon Echo, Microsoft Cortana and the Google Home, people increasingly prefer to interact with services by speaking rather than typing. Luckily IBM Watson also has Text to Speech and Speech to Text services. In this post we show how to hook these services together to provide a unified speech interface to Watson's capabilities.

In this blog we will build upon the existing SpeechToSpeech sample which takes text spoken in one language and then leverages Watson's machine translation service to speak it back to you in another language. You can try the application described here on Bluemix or access the code on GitHub to see how you can customise the code and/or deploy on your own server.

This application has only one page and it is quite simple from the user's point of view.

• At the top there is some header text introducing the sample and telling users how to use it. 
• The sample uses some browser audio interfaces that are only available in recent browser versions. If we detect that these features are not present we put up a message telling the user that they need to choose a more modern browser. Hopefully you won't ever see this message.
• In the original sample there are two drop down selection boxes which allow you to specify the source and target language. We removed these drop downs since they are not relevant to our modified use case.
• The next block of the UI gives the user a number of different ways to enter speech samples:
• There is a button   which allows you to start capturing audio directly from the microphone. Whatever you say will be buffered and then passed directly to the transcription service. While capturing audio, the button changes colour to red and the icon changes  - this is a visual indication that recording is in progress. When you are finished talking, click the button again to stop audio capture.
• If are working in a noisy environment or if you don't have a good quality microphone, it might be difficult for you to speak clearly to Watson. To help solve this problem we have provided you with some ample files hosted in the web app. To play one of these samples click on one of the buttons to play the associated file and use it as input.
• If you have your own recording that you can click on the  button and select the file containing the audio input that you want to send to the speech-to-text service.
• Last, but not least, you can drag and drop an audio file onto the page to have it instantly uploaded
• The transcribed text is displayed on an input box (so you can see if Watson is hearing properly) and sent to either the translation service (in the original version) or the conversation service in our updated service. If there is a problem with the way your voice is being transcribed, see this previous article on how to improve it.
• When we get a response from the conversation or translation service we place the received text on an output text box and we also call the text-to-speech service to read out the response and save you the bother of having to read.

I know that you want to understand what is going on under the covers so here is a brief overview:

• The app.js file is the core of the web application. It implements the connections between the front end code that runs in the browser and the various Watson services. This involves establishing 3 back-end REST services. This indirection is needed because you don't want to include your service credentials in the code sent to the browser and because your browser's cross site script protections will prohibit you from making a direct call to the Watson service from your browser. The services are
• /message - this REST service implements the interface to the Watson Conversation service. Every time we have a text utterance transcribed, we do a POST on this URL with a JSON payload like {"context":{...},"input":{"text":"<transcribed_text>"}}. The first time we call the service we specify an empty context {} and in each subsequent call we supply the context object that the server sent back to us the last time. This allows the server to keep track of the state of the conversation.
  Most conversation flows are programmed to give a trite greeting in response to the first message. To avoid spending time on this the client code sends initial blank message when the page loads to get this out of the way.
• /synthesize - this REST service use used to convert the response into audio. All that this service does to convert a get on http://localhosts:3000/synthesize?voice=en-US_MichaelVoice&text=Some%20responsevoice=en-US_MichaelVoice&text=Some%20response into a get on the URL  https://watson-api-explorer.mybluemix.net/text-to-speech/api/v1/synthesize?accept=audio%2Fwav&voice=en-US_MichaelVoice&text=Some%20response this will return a .wav file with the text "some response" being spoken in US English by the voice "Michael". 
• /token - the speech to text transcription is an exception to the normal rule that your browser shouldn't connect directly to the Watson service. For performance reasons we chose to use the websocket interface to the speech to text service. At page load time, the browser will do a GET on this /token REST service and it will respond with a token code that can then be included in the URL used to open the websocket. After this, all sound information captured from the microphone (or read from a sample file) is sent via the websocket directly from the browser to the Watson speech to text service.
• The index.html file is the UI that the user sees. 
• As well as defining the main UI elements which appear on the page, it also  includes main.js which is the client side code that handles all interaction in your browser.
• It also includes the JQuery and Bootstrap modules. But I won't cover these in detail.
• You might want to have a closer look at the client side code which is contained in a file public/js/main.js:
• The first 260 lines of code are concerned with how to capture audio from the client's microphone (if the user allows it - there are tight controls on when/if browser applications are allowed to capture audio). Some of the complexity of this code is due to the different ways that different browsers deal with audio. Hopefully it will become easier in the future. 
• Regardless of what quality audio your computer is capable of tracking, we down sample it to 16bit, mono at 16 Khz because this is what the speech recognition is expecting.
• Next we declare which language model we want to use for speech recognition. We have hardcoded this to a model named "en-GB_BroadbandModel" which is a model tuned to work with high fidelity captures of of speakers of UK English (sadly there is no language model available for Irish English). However, we have left in a few other language models commented out to make it easy for you if you want to change to another language. Consult the Watson documentation for a full list of language models available.
• The handleFileUpload function deals with file uploads. Either file uploads which happen as a result of explicitly clicking on the "Select File" button or upload that happen as a result of a drag-and-drop event.
• The initSocket function manages with the interface to the websicket that we use to communicate to/from the speech_to_text service. It declares that the showResult function should be called when a response is received. Since it is not always clear when a spaker is finnished talking, the text-to-speech can return several times. As a result the msg.results[0].final variable is used to deremine if the current transcription is final. If it is an intermediate result, we just update the resultsText field with what we heard. If it is the final result, the msg.results[0].alternatives[0].transcript variable is also used as the most likely transcription of what the user said and it is passed on to the converse function.
• The converse function handles sending the detected text to the Watson Conversation Service (WCS) via the /message REST interface which was descibed above. When the service gives a response to the question, we pass it to the text-to-speech service via the TTS function and we write it on the response textarea so it can be read as well as listened to.
• In addition there are many other files which control the look and feel of the web page, but won't be described in detail here e.g. 
• Style sheets in the /public/css directory
• Audio sample files in the /public/audio directory
•  Images in the public/images directory
• etc.

Anyone with a knowledge of how web applications work, should be able to figure out how it works. If you have any trouble, post your question as a comment on this blog.

At the time of writing, there is an instance of this application running at https://speak-to-watson-app.au-syd.mybluemix.net/ so you can see it running even if you are having trouble with your local deployment. However, I can't guarantee that this instance will stay running due to limits on mypersonal Bluemix account.

Combining other services with IBM Watson Conversation Service

It is becoming increasingly popular to offer an interface to computer applications which resembles the way that we converse with a fellow human. The IBM Watson Conversation Service is an excellent way to program such an interface because it allows the developer an easy way to specify the conversation flow and is also very good at doing fuzzy matching on input text to guess what the user is really trying to find out. However, the graphical way that conversation flows are specified doesn't allow the user to make calls to external services in order to get information to be included in the reply.

People often need to call external services to get the information that their users are looking for and so in this article I describe a simple sample application written by myself and my colleague David Curran which shows a common pattern whereby the conversation service provides a template response along with parameters which can be used by the calling application to retrieve the necessary information to give the end user the answer that they are looking for.

This pattern is useful in a lot of different situations, but we will use a fictitious application of where people want to use a conversational interface to track their parcels. We will leverage the simple conversation application as a starting point to minimise the amount of work. You can either download that sample and follow the steps below to add the interface to the conversation agent, or if your prefer you can download the completed example from our GitHub repository.

Adding the Parcel Intent to the conversation

In order to modify the conversation agent to handle parcel requests, you first need to add a parcel intent to the list of intents. The original sample contains 25 intents which is the maximum allowed with the free plan, so you will need to delete one of the existing intents. I deleted the weather intent since it is not being used and then I added a parcel intent with a few sample inputs as you can see below,

The next step is to add a node to the dialog to specify how parcel queries are to be dealt with. Our logic is quite simple. If a number is detected in the input we assume that this is the parcel number so we set a context variable parcel_num with this value and then we send back a response message with placeholders where the parcel location should be inserted. However, if no number is detected in the input stream, we simply reply saying that they need to supply us with a parcel number. For simplicity sake we won't consider holding context from one question to the next.

Implementing the dummy parcel lookup service

We don't want to use a real parcel lookup service for this sample, because when testing we won't know the parcel number for parcels in transit. Instead we will implement a very simple lookup service.

To implement the parcel lookup service you need to add the following function near the end of app.js
 what this does is respond to get requests on /api/parcel and respond with one of the sample location names e.g. requesting http://localhost:3000/api/parcel?parcel_num=6 will return the string "Buckingham Palace". Just to illustrate how we should deal with errors, we have implemented the rule that if the parcel number is divisible by 13 it will return a status code 404 and an error message saying that the parcel number is unlucky.

/**
 * A dummy parcel tracking service
 */
 app.get('/api/parcel', function(req, res) {
   var parcel_num = parseInt(req.query.parcel_num);
   if (!req.query.parcel_num || isNaN(parcel_num)) {
     return res.status(400).end("Not a valid parcel number "
                                  +req.query.parcel_num);
   }
   if (0 == (parcel_num %13)) {
     return res.status(404).end("We can't find parcel number "
                                  +parcel_num+" it is unlucky!");
   }

   var locations = [
     'Anfield', 'Stamford Bridge', 'Old Trafford', 'Parkhead',
     'Heathrow Airport', 'Westminister, London', 'Buckingham Palace',
     'Lands End, Cornwall', 'John O\'Groats'
   ];

   parcel_num = parcel_num % locations.length;
   var location  = locations[parcel_num];
   res.end(location);
});

You should experiment with this service and/or customise it before moving on to the next steps.

Recognising a parcel location request and filling in the details

The main code  modification we need to do is in the app.post('/api/message',  function in app.js. However we first need to do some housekeeping changed due to the fact that we will be using the requestify library.

Add the following line to the dependencies section of package.json:

    "requestify": "^0.2.5",

Then add this line near the top of app.js

var requestify = require('requestify');

The nub of the code is contained in the function below. You should paste this into app.js to replace the call to conversation.message which is around line 56 of the original file.

  // Send the input to the conversation service
  conversation.message(payload, function(err, data) {
    if (err) {
      // the conversation service returned an error
      return res.status(err.code || 500).json(err);
    }
    var parcel_num = data.context.parcel_num;
    if (data.intents && (data.intents.length>0) && data.intents[0].intent
                  && (data.intents[0].intent === 'parcel') && parcel_num) {
      var server = 'localhost';
      var port = process.env.PORT || process.env.VCAP_APP_PORT || 3000;
      var url = 'http://' + server + ':' + port +'/api/parcel?parcel_num='+parcel_num;
      requestify.get(url)
        .then(function(response) {
          var location = response.body;
          data.output.text[0] = data.output.text[0].replace( /\{0\}/g, location);
          return res.json(data);
        })
        .catch(function(err){
          data.output.text[0] = "Parcel lookup service returned an error: "+err.body;
          return res.json(data);
        });
    } else {
      return res.json(data);
    }
  });

The original code did nothing other than calling the updateMessage function before passing the data received from the Conversation service back to the UI layer. However, the updateMessage function didn't do anything useful so we can delete it and instead we will call our dummy parcel location service to find the location of the parcel whose number appears in the context variable.

If the http call succeeds we assume that we have a good location and we replace any placeholder {0} strings in the message received from the conversation service with this location. If we get an error status from the http call, we replace the entire string received from the conversation service with a message saying we failed to locate the parcel.

Summary

You have a conversation service which can reply to questions such as "where is my parcel number 543210" with details of where the parcel is located. It is currently only using a toy implementation which pseudo-randomly selects locations in the UK. However, it should be relatively easy to extend it to any real parcel tracing service you want. In fact, the methods used can easily be applied to interfacing with any 3rd party service.

You can go to https://github.com/bodonova/conversation-parcel to download the complete working sample.

Sending an Email from Watson Conversation Service

Sending emails is a simple way to connect a chatbot to a business process. This post shows how to extend the Watson Conversation Sample Application to get it to send email from a Gmail account. We will use the nodemailer library and a slightly modified version of the code from w3 schools to help us.

The first step is to download the original sample application from GitHub. Follow the instructions in the Readme file in the repository to get this application running on your local machine and/or on the BlueMix service. Make sure you have the original application working correctly before you go on to make any changes.

Modify the code below to reference the username and password of an Gmail account you have access to (or create a new account) and then add it near the top of the app.js file.

var nodemailer = require ('nodemailer');

var transporter = nodemailer.createTransport({  
  service: 'Gmail',  
  auth: {  
   user: 'chatbot@gmail.com',  
   pass: 'secretpassword'  
  }  
 });  
 var mailOptions = {  
  from: 'chatbot@gmail.com',  
  to: 'name@email.com',  
  subject: 'Sending Email from the chatbot',  
  text: 'That was easy!'  
 }  

If you use a gmail email and two factor authentication in Gmail you have to get an app password for your gmail address. Google are worried you will share your Gmail password with people so they give you a special password just for your app that has limited powers.

Next update the Watson application's package.json to say that you want to install the nodemailer package and force the server to use a version of node.js that supports nodemailer i.e.:

  "dependencies": {
...,
  "nodemailer": ">0.3.x"
  },
...
  "engines": {
    "node": ">= 6.9.x",
    "npm": "> 5.30.x"
  }

Having done this, the next step is to change the conversation flow in the Watson Conversation Service so that when the user says ‘send an email’, we will set a context variable called 'email' to say an email should be set. (A production version would probably send different emails to different addresses depending upon context, but we will stick to a simple example for now)

If your WCS has an intent that you want to send an email. This contains example user utterances like

#sendEmail
Can you email for me?
email dave important info
Ping dave an email to tell him to do important business things
Send an email to find my package
Send an email reminder message
Message Dave a mail to tell him the stuff he needs to do

In WCS dialog tab we want a node that recognises this intent and sets a context variable to signal to the node application to send an email.

In the json editor of the dialog node add a context variable for the email.

{
  "context": {
    "email": "TRUE"
  },
  "output": {
    "text": {
      "values": [
        "Ill email david now"
      ],
      "selection_policy": "sequential"
    }
  }
}

Next, you need to change app.js to say that when this ‘send an email' context variable is send it. At the start of the function updateMessage add in the following code check to see if the context variable email from WCS' response has been set and send an email if it has.

if(response.context.email){
    response.context.email=null;//set send variable to null so we don't send email every time
  transporter.sendMail(mailOptions, function(error, info){

    if (error) {
      console.log(error);
    } else {
      console.log('Email sent: ' + info.response);

    }
  });  
}

You can download the complete sample from here

This code just sends the same email every time. In practice you would probably have to include some information from the conversation in the mail. Usually this would something from context variables. You would add in the email information the user had told the chatbot such as the value in response.context.query

I am not a node developer and this code needs some error correction and security checks added to it. But for a quick demo it should allow you to show your chatbot emailing.