Deleting all documents from a Watson Discovery collection

Sometimes when you are cleaning up a collection, you might want to delete all of the documents. It is tedious to do this by clicking on the files in the UI. Therefore I wrote a simple script del_docs.ipynb to do this for you.

To run this script, edit the panel containing the collection details and then jus run the notebook.

By default the script will delete all documents, if you want to only delete a subset of documents then alter the query variable to specify the selection formula for the documents you want to delete.

Automatically emailing a copy of conversation history

Many chatbots offer the option of emailing the user a copy of the conversation history. Traditionally this has been done by writing code server side, but with the latest version of the web widget associated with Watson Assistant it is really easy to implement this with a small bit of client side code.

The code to implement this can be downloaded from GitHub. What you need to do is:

1. Place the code contained in the history_template.html file somewhere in the header part of your web site's page. 
2. Modify the INTEGRATION_ID, REGION_ID and INSTANCE_ID variables to match your Watson Assistant instance.
3. Create an account on elasticmail.com if you don't already have one. Then insert the SMTP_HOST, SMTP_USERNAME, SMTP_PASSWORD and FROM_ADDRESS in the javascript code.

The code is fairly easy to understand, but here are a few pointers:

• The start and end of the code block is the same as the web widget which is provided for you by the Watson Assistant UI. In fact accessing this widget is the easiest way to get the correct values for the integrationID:, region:, and serviceInstanceID: variables.
• The code registers a number of event handlers:
• The histHandler() function gets called when the chat window is opened. The event passed contains the history which has been stored client side. This is used to construct the initial email contents and store it in the global variable historyText.
• The msgSent() and msgReceived() functions are called when a message is sent by the end user and by Watson respectively. They update the global variable historyText.
• Last, but not least the msgSent() event handler gets called whenever the chat window is closed/minimised. This calls the sendEmail() function described below.
• The sendEmail() function implements the actual sending of the email. The documentation for this javascript library suggests that any arbitrary SMTP server can be used, but I found that it wouldn't work with any SMTP agent apart from smtp.elasticemail.com. The value of the From: variable indicates where the email will appear to be coming from - this will have to be the email address you used to create the account on ElasticMail. Be careful what address you use - this is where any replies will be sent.

If you want to customise the look and feel of the chat history emails, edit the variable baseHistoryText and/or change the watsonMsg() and userMsg() functions.

I hope you find this sample useful. As always this code is provided on a As-Is basis. No warranty or support is offered.

Video Tutorials on chatbot Making

I have started making videos explaining the chatbot making process. These are not Watson specific. More the steps needed in general.

Understanding Intents and Entities

One of the most important things that any conversational agent needs to do is to figure out what intent(s) and entities are contained in the user's input. An intent is what the user wants to do and the entities are the things that they want to do it with or too. This can be confusing so this post tries to explain with a simple example.

To illustrate we will consider a chatbot written for a second hand electronics store. The samples of expected user input might be something like:

1. I want to sell my old iPhone
2. I need a new laptop
3. I am interested in buying an new iPad

In examples 2 and 3 the user's intent is to buy, while in example statement number 1 the intent is to sell. When designing a conversational agent you define the intents by giving examples of what you expect users with this intent to say. In a real system you would need to give many more examples to give Watson a better chance of  guessing the users' intent, but this is enough to illustrate the way it works..

There are different ways to specify entities. The most common one is to give samples of the entity values. For example, if we had an entity @device and the possible values were 'phone', 'tablet' and 'laptop' - we would specify the entity as shown below by giving examples of what the user might say for each type of entity.

The second way of specifying an entity is with a regular expression. This is useful in cases where you want to catch an email address or phone number. In this cases it is not feasible to exhaustively list all possible inputs, but the rules for what an email address or phone number look like are easy to specify in a regular expression.

The third option for entities is to use one of the predefined system entities. For example, you might use the @sys-date entity a simple way to capture mentions of a date without having to go to the trouble of specifying a complex regular expression. This also has advantages such as when the user types 'tomorrow' the entity extracted is the correct date for the day after they typed it.

Many developers of chatbots don't realise that you can combine entities with intent examples to make them more powerful. If you specify your examples like this:

1. I want to sell my old @device
2. I need a new @device
3. I am interested in buying a new @device

This saves you the trouble of repeating the same sentence for each type of device that the users might want to buy or sell.

Convert Watson csv's to RASA yml

Watson allows you to export the ground truth (questions and intents) RASA expects data in a slightly different format.
Watson is in the form

Question, Intent
Can I buy a sandwich?, #buy_sandwich

RASA in the form

## intent:buy_sandwich
- Can I buy a sandwich?

This is yml format but RASA calls it .md. The code for this conversion from Watson format to RASA is below

import pandas as pd

 questions = pd.read_excel(filePath,names=['Intent','Question'])

file = open("rasaoutput.md","w", encoding="utf-8")  

 

for x in labels:
    #print the intention name in rasa way
    file.write("## intent:"+x+"\n")
    #then print every question in the dataset with that label
 
    y=questions[questions['Intent'].str.contains(x)]['Question']
    #change series into an array
    z=y.values
    i = 0
    while i < len(z):
        file.write(str("- "+z[i]+"\n"))
        i=i+1

    -David

Migrating your Watson Assistant workspace to WebHooks

In the last post we examined the differences between webhooks and the old way (sometimes called web actions) that IBM Watson assistant called REST functions through the use of IBM Cloud Functions. In this post we will look at a simple example of migrating from one to the other.

There is a tutorial on DeveloperWorks which guides you through all of the steps to connect your Watson Assistant skill to the Wikipedia API using the old style mechanism. In this blog post we will assume that you have already gone through the steps in the original tutorial and we will describe how you can convert your workspace to use the newly released webhooks feature.

The first step in the migration is that  you need to make sure that the Cloud Function you created to lookup Wikipedia call able to be called externally.

If the function definition is associated with a resource group, the security model will make it hard to be called, so you need to ensure that your function definition it is associated with a Cloud Foundry space. You can tell if your selected namespace is IAM based or Cloud Foundry based because of the drop-down selector at the top of the page will say (CF-Based) e.g.:

If your function is defined in a IAM resource group, the easiest way to move it is to create a new function (with the exact same code) in a Cloud Foundry space i.e. switch to the new space with the drop-down and then create the function as described in the original tutorial.

After you have saved the Action you should try it out by changing the object_of_interest to different things you might be interested in looking up and then see what the function returns.

Since WebHooks doesn't interact directly with Cloud Functions as such, you will need to ensure that your action is turned into a WebAction which can be called by any REST client. To do this, click on the EndPoints link in the left margin. This will give you the option to make your action invokable by a REST URL.

Once  you do this, you will see a curl command which can be used to invoke the web action. Initially the screen only shows API-KEY rather than the actual API key assigned to you. Click on the eye icon on the right to display the fully correct curl command.

If you have curl installed you can copy this command and execute it in your command line window, However, you will get an error because you haven't given any input parameters. To solve this add more command line options to specify the object of interest and the fact that the data you are supplying in in JSON format. e.g.:

curl -u 3a686c56-12fc-4bd9-8a08-55317fec468d:CE4A88p1qGYV69dF43iVENNn3Ok6DHdtlYz7tlrCh0yG7aNRvUgzcHHBNJxi15z9 --header "Content-Type: application/json"  --data "{\"object_of_interest\": \"love\"}" -X POST https://us-east.functions.cloud.ibm.com/api/v1/namespaces/brian_odonovan_bod-space/actions/Assistant-Functions/Wikipedia-Lookup?blocking=true

If you prefer using another tool like POSTMAN, you can easily convert this command to suit. The one thing you need to be aware of is the fact that the authorisation you supply with the -u parameter to the curl command consists of two parts - the part before the colon is effectively a username and the part after the colon is the password.

Once you have verified that you WebAction is working correctly, you next need to change the Watson Assistant skill to use webhooks when calling Wikipedia. You do this by clicking on the options tab when editing the dialog and then selecting the Webhooks option on the left (it should be selected by default) and then entering details of the URL you want to call, what credentials to use and any other headers you want to pass to the function.

You can use the URL and credentials from the curl command that you got from the web actions page described above. You might be slightly worried that there is a single URL assigned to a  skill because in some cases where you might need to access services from different sites. There are ways of getting around this limitation, but I won't describe them here since our use case doesn't need to connect to multiple services. The next blog post in this series will describe in detail how you can connect to multiple REST services from a single WA workspace..

The Dialog node which implements the interface to Wikipedia through Webhooks will be significantly different from the old one, therefore I suggest that you either delete or disable the old node. For example you could rename the node to Old Wikipedia and disable it my changing the match condition to false as illustrated below.

Now you have to create a new Dialog for calling the webhook. You should call the node Wikipedia or something similar and make sure it is activated whenever the #tell_me_about intent is detected. Next click on the 'customize' icon and this will give you an option to turn on Webhooks for this node.

As soon as you close the customize dialog you will see that you see additional UI elements which parameters you would like to pass to the REST call and what context variable you would like to use to store the response.

You will also see that the node has been converted into a multi condition response node and it pre-configures two output slots for what to say when your REST call succeeded (i.e. when the context variable has been set) or when the variable wasn't set (which probably indicates a network error or something similar).

You can use the same responses as in the in the original tutorial since the format of the response won't have changed. You can now test your application and see that it behaves more or less as before.

There are two things that you should note about the way that Watson Assistant Webhooks work:

1. We specified that you add a parameter named object_of interest and set its value to the contents of the @object_of_interest entity (lets assume that you asked "what is love" so the value will be "love" ).
   
   Normally when people say that they are adding parameters to a POST call they mean that they are adding a header with the value "object_of interest: love". However, this is not what Watson Assistant does. Instead it sends a JSON body with each of the parameter values e.g. {"object_of_interest": "love"}.
   
   This is actually a better thing to do, but make sure you don't get confused by the terminology in the documentation.
2. Watson Assistant tells you that it stores the response from the REST call in the context variable you specify, but this is not exactly what it does. While the Webhooks functionality is not totally tied to the Cloud Functions, it does make certain assumptions based upon the way Cloud Functions operate.
   
   The response from a call to a Cloud Function will contain lots of information about the call other than just the response from the REST service called. It look something like: 

{

   "activationId": "xxx",
   "end": <time_stamp>,
   "start": <time_stamp>,
   "response": {
      "result": { ...},
      "status": "success",
      "success": true
   },

   ...
}

When you IBM Cloud Function returns, the data returned by the REST service is contained in the response.result part of the JSON structure retuned. If you are not using Cloud Functions, make sure you follow this convention because Watson Assistant will be expecting it. Similarly, you should also set the response.sucsess variable to the value true because otherwise Watson Assistant will assume that the call has failed.

What is the benefit of the new webhooks feature in Watson Assistant

When building an AI chatbot it is impossible to incorporate all knowledge directly in your skill. As a result developers often find themselves wanting to call external functions to answer certain queries. IBM has responded to this requirement by supporting the calling of cloud functions from within a Dialog node in Watson Assistant.

While developers have found this useful, they have also complained that it is inflexible and not so easy to use. To answer these complaints,  IBM has recently released a new feature called webhooks. This feature was available in limited Beta for several months, but has just been released generally so now is a good time to look at it.

This table summarises the differences between the two mechanisms:

Aspect	webhooks	old way
URL Flexibility	With webhooks you can call any arbitrary URL. This means that you are not tied to using IBM Cloud Functions as an intermediate layer, Of course Watson Assistant will always make a POST call to your URL and supply the parameters in JSON. If the REST API you want to call does not accept this, you will need some mechanism to transform the call. However, you are free to choose any transformation tool that you want.	With the old mechanism, you could only call a Cloud Function which is defined in the same environment as the Watson Assistant instance which is doing the calling. This was quite restrictive and although it worked OK with Cloud Foundry based authentication, it was not really compatible with the new IAM style resource group authentication currently used in the IBM cloud.
UI Assistance	There is a UI to guide you in defining the authorisation and other parameters for your call to a webhook. This makes it quite user friendly.	The way that you specified that a REST call should be made is by editing the JSON response from a node to include an action parameter. Apart from the documentation there was no assistance to developer to define this correctly.

Now that we have compared the two mechanisms, our next blog post will look at a simple example of migrating from one to the other.