Session 6 Improving Conversational Solution Lab Instructions 181112
User Manual:
Open the PDF directly: View PDF 
.
Page Count: 35
Lab Instructions
Laurent Vincent
Session 6: Improving
IBM Watson Assistant
Page 2 of 35
Content
Let’s get started ...................................................................................................................................... 3
1. Overview ................................................................................................................................ 3
2. Objectives .............................................................................................................................. 3
3. Prerequisites .......................................................................................................................... 3
4. Scenario ................................................................................................................................. 4
5. What to expect when you are done ................................................................................... 4
Improving understanding ...................................................................................................................... 5
6. Overview page ...................................................................................................................... 5
7. User conversations page ..................................................................................................... 8
Conversation Sample Application ........................................................................................................ 9
8. Install locally your test application ...................................................................................... 9
9. Add a deployment ID .......................................................................................................... 12
10. Start your local application ............................................................................................. 13
11. Test your local application ............................................................................................. 14
Sending utterances and generating logs with Watson Studio .......................................................... 17
12. Create a Watson Studio instance ................................................................................. 17
13. Create Watson Studio resources .................................................................................. 18
Improve Conversation ......................................................................................................................... 23
14. Navigate on Overview page........................................................................................... 23
15. Create a test-experimental workspace ........................................................................ 23
16. Selecting data source / Deployment ID ....................................................................... 24
17. Adding an entity value or synonym............................................................................... 25
18. Correcting an intent ......................................................................................................... 27
19. Irrelevant input ................................................................................................................. 30
20. Test your conversation updates .................................................................................... 31
Test your improvements with Watson Studio ................................................................................... 32
21. Prerequisite : create a cloudant service....................................................................... 32
22. Create Watson Studio resources .................................................................................. 34
Page 3 of 35
Let’s get started
1. Overview
The IBM Watson Developer Cloud (WDC) offers a variety of services for developing
cognitive applications. Each Watson service provides a Representational State
Transfer (REST) Application Programming Interface (API) for interacting with the
service. Some services, such as the Speech to Text service, provide additional
interfaces.
The Watson Assistant service combines several cognitive techniques to help you
build and train a bot - defining intents and entities and crafting dialog to simulate
conversation. The system can then be further refined with supplementary
technologies to make the system more human-like or to give it a higher chance of
returning the right answer. Watson Conversation allows you to deploy a range of
bots via many channels, from simple, narrowly focused bots to much more
sophisticated, full-blown virtual agents across mobile devices, messaging platforms
like Slack, or even through a physical robot.
The illustrating screenshots provided in this lab guide could be slightly different
from what you see in the Watson Assistant service interface that you are using. If
there are colour or wording differences, it is because there have been updates to the
service since the lab guide was created.
2. Objectives
Watson Conversation Service provides features to analyse its usages.
In this lab, you will:
• Learn how to create a Node.js application running locally
• Learn how to improve the Watson Conversation service using Node.js
• Learn how to leverage Watson studio to test your service
3. Prerequisites
You must have install on your workstation Node.js and Cloud Foundry as describes
in the prerequisite document
• Session 5 – Building a dialog
• The instructor provided you the link to get labs content. You may download
each file individually.
Page 4 of 35
IBM Cloud URLs per location:
Location
URL
US
https://console.ng.bluemix.net/
UK
https://console.eu-gb.bluemix.net/
Sidney
https://console.au-syd.bluemix.net/
Germany
https://console.eu-de.bluemix.net/
4. Scenario
Use case: A Hotel Concierge Virtual assistant that is accessed from the guest
room and the hotel lobby.
End-users: Hotel customers
5. What to expect when you are done
At the end of session, you should have
1. one Workspace for production
2. one Workspace for test
3. a simple node.js application to start testing your conversation
4. a Watson studio service with 2 notebooks
5. the right understanding of the improve capabilities.
Page 5 of 35
Improving understanding
The Improve component of the Conversation service provides a history of
conversations with users. You can use this history to improve your bot's understanding
of users' inputs.
While you develop your workspace, you use the Try it out panel to verify that it
recognizes the correct intents and entities in test inputs, and make corrections as
needed. In the Improve panel, you can view information about actual conversations
with your users and make similar corrections to improve the accuracy with which
intents and entities are recognized
1. Click on the hamburger menu (top left) and click on Improve
The User Conversation page of the Improve component provides a summary of
interactions with your bot, such as Statistics and chat logs for the last 90 days are
provided, with the newest first. You can view the traffic for a given time period, as well
as the intents and entities that were recognized in user conversations. These statistics
represent external traffic that has interacted with your bot; they do not include
interactions that use the Try it out panel in the tool.
6. Overview page
1. Click on overview tab
Page 6 of 35
Use the time period control to choose the period for which data is displayed. This
control affects all data shown on the page: not just the number of conversations
displayed in the graph, but also the statistics displayed along with the graph, and the
lists of top intents and entities.
You can choose whether to view data for a single day, a week, a month, a quarter, or
a year. In each case, the data points on the graph adjust to an appropriate
measurement period. For example, when viewing a graph for a day, the data is
presented in hourly values, but when viewing a graph for a week, the data is shown
by day. A week always runs from Sunday through Saturday. You cannot create custom
time periods, such as a week that runs from Thursday to the following Wednesday, or
a month that begins on any date other than the first.
You can select View logs to open the User Conversations page with the date range
filtered to match the time period that you have selected for the Overview page.
Depending on your plan and the date range that you selected, it is possible that the
User Conversations page will show no data.
Page 7 of 35
While viewing the graph, you can click on an individual data point to see the numeric
value, as shown here:
You can use the View logs link to open the User Conversations page with the date
range filtered to match the data point.
In addition to the graph, four statistics related to the displayed data are shown:
1. The total number of conversations that took place during this time period
2. The average messages per conversation is the total messages divided by
the total conversations during the selected date range.
3. The maximum number of conversations for a single data point within the
time period
4. The number of messages not managed correctly.
You also see the intents and entities that were recognized most often during the
specified time period. By default you see the top three of each, but you can change
that to a larger number.
Intents are shown in a simple list. In addition to seeing the number of times an intent
was recognized, you can click on the intent name to open the User Conversations
page with the date range filtered to match the data you are viewing, and the intent
filtered to match the selected intent.
Entities are shown in a simple list. Click on it to open the User Conversations page
with the date range filtered to match the data you are viewing and the entity filtered to
match the selected entity.
The page shows when the data that it displays was last updated. You can select
Refresh data, at the top of the page, if you think that newer data might be available.
Page 8 of 35
7. User conversations page
As you didn’t do any interaction with external application, the page should be empty:
You can also filter the chat logs by:
• Search user statements... - Type a word in the search bar. This searches the
user's input, but not your bot's reply.
• Intents - Select the drop-down menu and type an intent in the input field, or
choose from the populated list. You can select more than one intent, which
filters the results using any of the selected intents.
• Entities - Select the drop-down menu and type an entity name in the input field,
or choose from the populated list. You can select more than one entity, which
filters the results by any of the selected entities. If you filter by intent and entity,
your results will include the messages that have both values.
• Date range - Select the drop-down menu and choose Last 7 days, Last 30
days, Last 90 days (Default), or Custom. If you choose Custom, type the date
range or select it from the calendar. Chat logs history is dependent on your
service plan, as stated in the plan description.
Page 9 of 35
Conversation Sample Application
The Node.JS app you are going to install in your laptop is the conversation service in
a simple chat interface. It can be used as a basis for testing, improving and building
your own application.
8. Install locally your test application
1. Use GitHub to clone the repository locally, Go to
https://github.com/watson-developer-cloud/assistant-simple
2. Click on Clone or download button and Download ZIP
Page 10 of 35
3. Then select the right destination folder to save the assistant-simple-master.zip
file.
4. Several extracted files are hidden such as .env.template file. If you don’t see it in
the unzipped folder conversation-simple-master, you must enable this option
“show hidden file” on your laptop.
On IOS:
1. Open Terminal
2. Run the following script:
defaults write com.apple.Finder AppleShowAllFiles true
killall Finder
If you want to switch it back just change the true to false.
On windows: update folder settings
5. Copy or rename the .env.example file to .env
On windows:
1. Open Terminal
2. Go to the location of your project
On windows:
3. Run the following script:
Copy .env.example .env
On IOS:
4. Run the following script:
Cp .env.example .env
Page 11 of 35
6. Retrieve the credential of your skill. Go back to the skills tab
7. Click on 3 dots menu of your skill tile, click View API Details
then you get
here you will find all details such as Username, Password, Assistant ID required to
leverage Watson conversation APIs.
8. Open .env file
9. Paste the Workspace, User and password values (without the <> marks) into the
corresponding values.
# Environment variables
WORKSPACE_ID= <Workspace ID>
# You need to provide either username and password
ASSISTANT_USERNAME=apikey
ASSISTANT_PASSWORD=<password>
ASSISTANT_URL=https://gateway.watsonplatform.net/assistant/api
10. Save .env file
Page 12 of 35
9. Add a deployment ID
For a Watson Assistant Service that includes multiple workspaces, it could be useful
to use utterance data from one workspace to improve another workspace within that
same service.
Note: If you are a Assistant Premium user, your premium instances can optionally be
configured to allow access to log data from workspaces across your different premium
instances.
As an example, you may have both a Production workspace and a Development
workspace in your Assistant service. When working in the Development workspace,
you can filter utterances based on the Deployment ID for the Production workspace,
so that you are using Production workspace utterances to improve your Development
workspace.
You might also want to filter data regarding the channel, environment or user group.
To make this possible, you are going to add the deployment ID in the message you
send to WCS. This is a property inside the metadata object in your context.
1. Edit app.js file to get this result
2. Look for the playload definition
3. Add the deployment ID definition, around the line 50
var newcontext = req.body.context || {};
var newmetadata = {'deployment':'ChatbotProduction'};
newcontext['metadata'] = newmetadata;
Page 13 of 35
4. Comment the line context: req.body.context || {} and add the code like this
// context: req.body.context || {},
context: newcontext || {},
5. Save and close the file
10. Start your local application
1. In a terminal/command window, issue a change directory command to navigate to
a top-level directory “assistant-simple-master” that will contain your Node.js
projects.
2. Install all dependencies of your demonstration application package, enter the
command :
npm install
3. Start your application, enter the command:
npm start
4. Point your browser to http://localhost:3000 to try out the app.
The chat interface is on the left, and the JSON that the Javascript code receives from
the Assistant service is on the right. Type any request to test your Chatbot. The system
understands your intents and responds. You can see the details of how your input in
the user input section was understood by examining the JSON data in the Watson
understands section on the right side.
Page 14 of 35
11. Test your local application
1. Test your application.
For example, if you type where is the pool, the JSON data shows that the system
understood the #hotel_location intent with a high level of confidence, along with
@hotel_amenity entity with the value pool.
2. In order to populate your data base, continue to test the application like you did in
the try it out panel.
To prepare the improvement, you will enter intents and entities unknown by Watson
Page 15 of 35
3. Type I would like an apple pie.
Watson identifies #eat intent but doesn’t identify any entity. #eat is a wrong
classification.
4. Type French and now to go out the Find restaurant branch.
5. Type Where can I grab something to eat?
Hotel_location intent could be a right classification , but we want to associate this
user’s example to #eat intent.
Page 16 of 35
6. Type I want an Iphone
Watson identifies the intent #General_Jokes. This example is not relevant in our
use case.
You can continue to test and submit input to Watson with sentences which make sense
for you.
Page 17 of 35
Sending utterances and generating
logs with Watson Studio
This chapter proposes a way to populate automatically the log of your Watson
assistant workspace by using Watson Studio. A notebook is provided to leverage the
WA Python SDK.
12. Create a Watson Studio instance
You can use also an existing instance of Watson Studio in the right region. Go to the
next step in this case.
1. Go back on your IBM Cloud Dashboard.
2. Create a new resource (create resource button)
Note: be sure to create the Watson studio instance and the Watson Assistant in the
same region.
3. Look for Watson studio
4. Then click on Watson Studio tile
5. Select US south / Dallas as region
6. Keep the default group and click Create
Page 18 of 35
7. Click Get Started
You are going to use Watson Studio to test the efficiency of your Ground Truth.
13. Create Watson Studio resources
1. Click on new project tile, you can use an existing project (with Jupyter Notebooks)
and just add a new notebook
2. Select the Standard tile and click Create Project.
Page 19 of 35
3. Name it Assistant Evaluation
4. Keep the default storage, click Create
5. Select Assets tab
6. Select Load tab from the data panel.
7. Upload utterances-for-generate-chat-logs.csv csv file created to populate your
logs.
Then your data set is available in your project
8. Click Add to project and Notebook
Page 20 of 35
9. Select From File tab, then browse
10. Select the notebook Sending utterances and generating logs.ipynb provided in the
Box
11. Select the right runtime Python 3.5 Free
12. At the bottom, Click Create Notebook
Page 21 of 35
13. Now you can follow the instructions in the notebook
Page 22 of 35
Once the notebook executed, the Improve page should look like this:
Page 23 of 35
Improve Conversation
Now, You are going to work with 2 workspaces:
The one where you built your dialog and where you have populated the logs. It
will be use to simulate the production environment
A new one , you will create, to apply your improvements which will simulate the
test environment.
14. Navigate on Overview page
1. Click on the hamburger menu (top left) and click on Improve
You should get some data. Be sure to select the right date.
2. Navigate to the page to work with the different features.
15. Create a test-experimental workspace
You are going to create a workspace to do the enhancements of your conversation
as we cannot do this in production..
1. Go back to the Skills tabClick on workspace
2. On your skill tile, click on the contextual menu and Download JSON to save it on
your local file system
3. Click Create New skill
Page 24 of 35
4. Go to Import skill tab, click choose JSON File, then select the file previously
saved on your local file system.
5. Click Import
6. Rename the new skill, as example ConvWks-180401-LV Test (use the tile menu
to do it) that the skill not assigned to an assistant.
Now your new test skill is available.
16. Selecting data source / Deployment ID
As you added the deployment ID ChatbotProduction in your message API calls, you
will be able to use utterances of your current skill from any other skills created in your
Watson Assistant service or filter the data according to the channel, the environment
or the user group.
1. Go to the Overview page of Improve feature of your Test experimental
workspace.
2. Select Data source then click Show deployment IDs
Now you have the possibility to select ChabotProduction data source.
3. Select ChatbotProduction data source.
Note: When switching to another data source, the Conversation service checks
utterances for an element called Deployment ID. Deployment IDs are unique
identifiers in the Conversation service API that you add to your message API calls.
Note: While Data source: now shows the source of the utterances you are using to
improve this skill, the top of the page still shows the workspace you are applying
changes to.
Page 25 of 35
17. Adding an entity value or synonym
1. At the bottom, on Top intents Select the #eat intent.
Watson opens the User conversations page and returns all records using #eat
intent.
You find that Watson didn’t identify Apple pie as entity. Apple pie could be added
as synonym of one of your entity to improve your chatbot.
2. To add an entity value or synonym, select the edit entities icon on the left of
the selected record.
3. Click Add entity
Page 26 of 35
4. Press shift and select the words apple and pie.
You must enter an entity to which the highlighted phrase will be added as a value.
Begin typing in the entry field, the list of entities will be populated. To add the
highlighted phrase as a synonym for an existing value, enter the entity:value.
5. Right now, enter restaurant and select @restaurant:French_restaurant entity
Page 27 of 35
then you will get this:
6. Click on Save button
if you look at the entity @restaurant the synonym apple pie has been added to french.
18. Correcting an intent
1. Click on User Conversations tab
2. As Search user statements, enter where can I grab
3. Look for the sentence, where can I grab something to eat
Page 28 of 35
4. You can see that Watson identifies the intent #hotel_locations and we want to
assig it to the intent #eat
5. To update an intent value, select the edit icon on the left of the selected
record.
6. From the drop-down list of intent, select #eat intent.
7. Click on Save button
8. You can repeat the previous steps for the user’s example Where can I grab a
hamburger ? and move it from #local_recommend to #eat intent.
Page 29 of 35
if you look at the intent #eat, the sentence has been added as utterance.
Note: you can review the whole conversation related to the selected record by using
open conversation.
Page 30 of 35
19. Irrelevant input
You can "Mark as irrelevant" to indicate that the input is not related to your application.
Inputs marked as irrelevant are stored in the workspace and are included as part of
the training data. They cannot be accessed or changed later in the tooling.
Note If you already have an intent for inputs that are out of scope or off topic, such as
#off_topic, you should delete that intent and test your workspace with the use of
"Irrelevant".
You are allowed 25,000 irrelevant examples per service instance.
You have seen that Watson identifies I want an Iphone as #General_Jokes intent.
To eliminate all ambiguity, you have some options :
• you must create new intent such as #buy, when a user wants to buy
something.
• Do not train on intent, which will not save this utterance as an example for
training
• Mark this sentence as irrelevant
It can be done by editing the intent
Page 31 of 35
20. Test your conversation updates
1. Return to your local application http://localhost:3000
2. Successively enter:
I want an apple pie
I want an Iphone
where can I grab something to eat
The virtual assistant should have the expected behaviors.
Page 32 of 35
Test your improvements with
Watson Studio
You are going to use Watson Studio to test the efficiency of your improvements by
comparing the behavior of your prod workspace and test workspace.
Go back to your Watson Studio instance
21. Prerequisite : create a cloudant service
One of the requirement of the next notebook is to create a Cloudant service
1. Go back to your IBM Cloud Catalog and create a cloudant instance or use a
existing one.
Page 33 of 35
2. Once created, Go to service credentials page (upper right menu)
3. Expand view credentials
4. Copy url , username, password
Page 34 of 35
22. Create Watson Studio resources
5. Go back to your Watson studio page, Open the Assistant Evaluation project
6. Select Assets tab
7. Click on New Notebook
8. Select From File tab, then browse
9. Select the notebook Watson Assistant Workspace Analysis with User Logs.ipynb
provided in the Box
10. Select the right runtime Python 3.5 Free
11. Click Create Notebook
Page 35 of 35
12. Now you can follow the instructions in the notebook
At the end of the execution, you should get the following graph.
You retrieve a delta for the intents hotel_location and eat . these are what you did
So it is expected.
You can see, also you update for the intent General_Agent_Capabilities. it means
that your update has some unexpected impacts which required investigation.
You can review the details in the last table:
It is up to you to determine if this is an improvement or not.
