Kofax Kapow User's Guide Users EN
KofaxKapowUsersGuide_EN
User Manual: Pdf
Open the PDF directly: View PDF .
Page Count: 532
Download | |
Open PDF In Browser | View PDF |
Kofax Kapow User's Guide Version: 10.2.0.1 Date: 2017-09-29 © 2017 Kofax. All rights reserved. Kofax is a trademark of Kofax, Inc., registered in the U.S. and/or other countries. All other trademarks are the property of their respective owners. No part of this publication may be reproduced, stored, or transmitted in any form without the prior written permission of Kofax. Table of Contents Chapter 1: Introduction.............................................................................................................................. 9 Getting Support...................................................................................................................................9 Chapter 2: Tutorials.................................................................................................................................. 11 Beginner Tutorials.............................................................................................................................11 Introduction.............................................................................................................................11 Robot Beginner's Tutorial...................................................................................................... 12 Kapplet Beginner's Tutorial....................................................................................................15 Type Beginner's Tutorial........................................................................................................ 17 Advanced Tutorials........................................................................................................................... 18 Branches, Robot States, and Execution Flow.......................................................................18 Looping Basics.......................................................................................................................20 Try Step................................................................................................................................. 24 Excel.......................................................................................................................................27 Data Conversion.................................................................................................................... 29 Patterns.................................................................................................................................. 31 Snippets................................................................................................................................. 37 Date Extraction - Simple Case..............................................................................................40 Date Extraction - Tricky Case............................................................................................... 41 API..........................................................................................................................................41 Chapter 3: Design Studio......................................................................................................................... 45 Introduction to Design Studio...........................................................................................................46 Robots.................................................................................................................................... 46 Snippets................................................................................................................................. 55 Variables and Types.............................................................................................................. 55 Libraries and Robot Projects.................................................................................................56 Naming policy........................................................................................................................ 57 Design Studio User Interface........................................................................................................... 58 Menu Bar............................................................................................................................... 59 Toolbar....................................................................................................................................61 My Projects View................................................................................................................... 62 Shared Projects View............................................................................................................ 63 Databases View..................................................................................................................... 63 Editors View........................................................................................................................... 64 Robot Editor........................................................................................................................... 65 3 Kofax Kapow User's Guide Type Editor.............................................................................................................................70 Text Editor.............................................................................................................................. 70 General Editing................................................................................................................................. 70 Types.................................................................................................................................................72 Type Attributes....................................................................................................................... 73 Step Actions and Data Converters.................................................................................................. 73 Patterns.............................................................................................................................................74 Expressions.......................................................................................................................................77 Experiment with Expressions................................................................................................ 78 Edit Expressions.................................................................................................................... 79 Projects and Libraries.......................................................................................................................80 Manipulate Robot Projects.................................................................................................... 81 Organize Robot Files.............................................................................................................82 Work with Shared Projects.................................................................................................... 82 Interact with Databases....................................................................................................................84 Map Databases......................................................................................................................84 Types and Databases............................................................................................................86 Database Warnings............................................................................................................... 86 Create Database Tables........................................................................................................ 87 Store Data in Databases....................................................................................................... 88 Robot Structure.................................................................................................................................91 Write Well-Structured Robots........................................................................................................... 92 Determine the Page Type................................................................................................................ 94 Use Tag Finders............................................................................................................................... 94 Tag Paths............................................................................................................................... 94 Tag Finder Properties............................................................................................................ 96 Configure Tag Finders........................................................................................................... 97 Submit a Form..................................................................................................................................97 Form Basics........................................................................................................................... 98 Determine the Step Action.................................................................................................. 100 Use the Loop Form Actions................................................................................................ 100 Upload Files......................................................................................................................... 101 Use the Context Menu on the Page View.......................................................................... 101 Loop Through Tags on a Page...................................................................................................... 102 Loop Through Tags with the Same Class...........................................................................102 Loop Through Tags with Different Classes......................................................................... 104 Loop Through HTML Pages...........................................................................................................105 First Page Links to All Other Pages................................................................................... 105 4 Kofax Kapow User's Guide Each Page Links to Next.................................................................................................... 106 Use Wait Criteria............................................................................................................................ 108 Extract Content from HTML........................................................................................................... 113 Extract Text.......................................................................................................................... 114 Extract Binary Data..............................................................................................................114 Use the Context Menu in the Page View............................................................................114 Perform Common Tasks...................................................................................................... 115 Extract Content From an HTML Table........................................................................................... 118 Handle Table Content Irregularities..................................................................................... 118 Handle Table Structure Irregularities................................................................................... 119 Local Files Usage in Robots.......................................................................................................... 119 Load an Excel Page from a Variable............................................................................................. 121 Extract Content from Excel............................................................................................................ 122 Extract Values from Cells.................................................................................................... 123 Extract a Sheet Name......................................................................................................... 124 Extract as HTML..................................................................................................................124 Test Cell Types in Excel.................................................................................................................125 Loop in Excel..................................................................................................................................126 Loop Over Sheets and Rows.............................................................................................. 127 Loop Over Merged Cells..................................................................................................... 128 Work with Variables in the Windows View.....................................................................................129 Open a Variable...................................................................................................................129 Modify a Variable................................................................................................................. 130 Work with JSON............................................................................................................................. 131 JSON Terminology............................................................................................................... 131 JSON MIME Type................................................................................................................132 JSON and Step Actions...................................................................................................... 132 JSON as a JavaScript Object............................................................................................. 134 Handle Errors..................................................................................................................................134 Error Handling Alternatives..................................................................................................135 Shortcuts for Common Cases............................................................................................. 136 At Target...............................................................................................................................138 Looping.................................................................................................................................139 Try Catch............................................................................................................................. 140 Identify Error Handling in Robot View................................................................................. 141 Create and Reuse Snippets...........................................................................................................142 Variables and Snippets........................................................................................................142 Snippet Best Practices........................................................................................................ 144 5 Kofax Kapow User's Guide Make Robust Robots......................................................................................................................145 Reuse Sessions..............................................................................................................................145 Modify an Existing Type................................................................................................................. 147 Configure Robots............................................................................................................................147 Show Changes from Default Robot Configuration.............................................................. 148 Migrate a Robot to a Different Browser Engine.............................................................................151 Migrating a Robot to the Classic Browser.......................................................................... 151 Migrating a Robot to the Default Browser...........................................................................151 Configure Variables........................................................................................................................ 152 Device Automation..........................................................................................................................153 Introduction to Device Automation...................................................................................... 154 Get Started with Device Automation................................................................................... 155 Reference to Automation Device.........................................................................................155 Use RDP Connection.......................................................................................................... 157 Device Automation Editor.................................................................................................... 158 Configure Automation Device..............................................................................................160 Finders in Device Automation............................................................................................. 169 Device Automation Steps.................................................................................................... 181 Automate Terminals............................................................................................................. 205 Use TLS Communication.....................................................................................................213 Expressions in Device Automation......................................................................................214 Variables in Device Automation...........................................................................................225 Limits in Numbers................................................................................................................226 Manage Remote Device...................................................................................................... 226 Debug Robots.................................................................................................................................228 Basic Debugging..................................................................................................................228 Debug from the Current Location in Design Mode............................................................. 230 Return to Design Mode from a Debugging Location...........................................................230 Use Breakpoints...................................................................................................................230 Single Stepping....................................................................................................................231 Step Into...............................................................................................................................231 Design Studio Settings................................................................................................................... 231 General.................................................................................................................................232 Text Files..............................................................................................................................232 Robot Editor......................................................................................................................... 232 Device automation............................................................................................................... 233 Local Databases.................................................................................................................. 233 Proxy Servers...................................................................................................................... 234 6 Kofax Kapow User's Guide Certificates........................................................................................................................... 235 Bug Reporting...................................................................................................................... 236 Management Consoles........................................................................................................ 236 Chapter 4: Management Console.......................................................................................................... 238 Introduction to Management Console Structure............................................................................ 238 Naming Policy...................................................................................................................... 239 Start the Management Console..................................................................................................... 239 Management Console Configuration and User Interface...............................................................240 Dashboard............................................................................................................................240 Kapplets............................................................................................................................... 242 Schedules.............................................................................................................................243 Repository............................................................................................................................ 250 Data......................................................................................................................................263 Logs......................................................................................................................................264 Admin................................................................................................................................... 267 Add Database Type............................................................................................................. 289 JMX................................................................................................................................................. 290 OAuth.............................................................................................................................................. 291 Supported Service Providers...............................................................................................291 Add Applications.................................................................................................................. 291 Add Users............................................................................................................................ 293 Write Robots........................................................................................................................ 296 Schedule Robots with Credentials...................................................................................... 297 Out of Band Applications.....................................................................................................298 Chapter 5: Kapow Kapplets................................................................................................................... 300 Building and Maintaining Kapplets................................................................................................. 300 Creating Kapplets................................................................................................................ 300 Using the Kapplet Studio.....................................................................................................301 Installing and Using Kapplets.........................................................................................................305 Invoking Kapplets.................................................................................................................305 Creating Email Notifications from Kapplets.........................................................................306 Scheduling Kapplets............................................................................................................ 306 Customizing Kapplet Branding....................................................................................................... 307 Chapter 6: Reference.............................................................................................................................. 308 Design Studio................................................................................................................................. 308 Step Action...........................................................................................................................308 Data Converters...................................................................................................................428 The Type Editor................................................................................................................... 469 7 Kofax Kapow User's Guide Creating and Deleting Tables.............................................................................................. 474 Protocols.............................................................................................................................. 474 Robot Libraries.....................................................................................................................475 Upload to Management Console.........................................................................................476 Other Topics.........................................................................................................................476 RoboServer..................................................................................................................................... 519 Start RoboServer................................................................................................................. 519 RoboServer Configuration................................................................................................... 523 RoboServer Configuration - Headless Mode...................................................................... 524 Management Console.....................................................................................................................527 Other Topics.........................................................................................................................527 Java API......................................................................................................................................... 529 Using Proxy Services..................................................................................................................... 530 Kapow Limitations...........................................................................................................................530 8 Chapter 1 Introduction Kofax Kapow is a platform for application integration and process automation. It can integrate applications that were not built to be connected and automate processes across such heterogeneous systems; cloud/ SaaS applications with premise systems, legacy systems with modern web applications, back office systems with partner websites. With our visual editor Design Studio, you click through the applications and data sources you want to integrate and create an automated workflow. In Kapow, these workflows are known as robots. As you build a robot, you are free to navigate through the applications as you integrate them. You can login to applications, extract data parts of a page, enter data into forms or search boxes, make menu selections, and scroll through multiple pages. Your robot can also access databases, files, APIs, and web services, exporting data from one application and loading it into another; transforming data as necessary along the way. Device Automation in Kapow helps you automate Windows and Java applications on your network computers. Device Automation replaces manual processes by controlling an application on a desktop or a terminal. See Device Automation for details. Once built, robots are uploaded to a repository in the Management Console. From here, they can be scheduled for batch-execution on the RoboServer or executed on-demand via Java or C# APIs, tailored REST services that are instantaneously available once the robots have been added to the repository, or exposed as special-purpose end-user web applications called Kapplets. The Management Console is also responsible for load balancing, failover, monitoring of RoboServer health and management of user roles and permissions. Getting Support Customer Support If you are having any kind of problems using Kofax Kapow, please go to http://services.kofax.com/support portal that can help you solve problems when using Kofax Kapow. In many of the Kofax Kapow applications, you can also send a bug report from within the application. To do this, select Report Bug in the Help menu. Please provide as much information as possible about the bug and what you did just before the bug occurred. Kofax Customer Portal and Knowledge Base Kapow customers who are active on maintenance also are entitled to obtain access to the Kofax Customer Portal, which includes solutions to commonly found problems, as well as a Knowledge Base containing implementation tips and tricks, and more. 9 Kofax Kapow User's Guide Support Policy For support policy, visit the Kofax End of Sale / End of Support Announcements page on the Kofax support portal. 10 Chapter 2 Tutorials The topics in this section contain links to video tutorials that help you perform different tasks in Kapow. On each tutorial page you can also find a transcript of the video. Note You need Internet connection to view the video tutorials. Beginner Tutorials This section contains several tutorials that provide an overview of Kapow as well as guide you through your first project in this product. Make sure to install and set up Kapow correctly before proceeding with these tutorials. Click the links to videos to play them. Note You need Internet connection to view the video tutorials. Introduction Introduction tutorial. This is the first of our beginner tutorials which will guide you safely through your first project with Kapow. In this first video, you will get an overview of the workflow involved when working with Kapow along with an introduction to the main program called Design Studio. Before watching these tutorials, make sure you have installed and set up Kapow correctly. Please follow the relevant parts of the Installation Guide. Kapow to put it simply is a platform which enables you to fully automate any process that you would be able to perform in a browser via your mouse and keyboard. Please sit back and watch as you are taken through the general procedure from idea to automated process. It all starts with an idea of a process you want to automate. In these Beginner Tutorials we want to automatically extract the most recent stories from a website called News Magazine. Our first step will be to check out the website. What exactly do we want? When we feel confident about what we want to achieve, we will open Design Studio, the program used to create the automated processes. The first time you open Design Studio you will get a welcome screen which links to this Beginner Tutorial, along with the rest of the documentation. Click OK and you will be able to see the main window of Design Studio. On the left side we have the projects view. Right now it contains only the default project which includes a collection of example files and a Tutorials folder where samples of the files we will be creating in these Beginner Tutorials can be found. 11 Kofax Kapow User's Guide Double clicking the file called Post.type in the projects view, opens it in the type editor, which is used to edit and create this kind of file. A type defines what kind of data can be stored in a variable of that type. If you are unfamiliar with types and variables, you can think of a variable as a bucket which can hold objects, like text or images, and the type can be thought of as the mold which produces that type of bucket. This particular type is designed to contain the information we will be extracting from the News Magazine website. In the very last tutorial we will get into the process of creating a type. The extraction of stories from News Magazine is performed by an automated process called a robot. Think of a robot as an automation of any process you would perform in a web browser. Double clicking the file called NewsMagazine.robot, also from the Tutorials folder, opens up the robot editor, which is used to edit and create robot files. The Robot View at the top of the editor displays the structure of the robot. Each step corresponds to an action performed by the robot. Going through the steps in the Robot View the robot loads News Magazine, navigates to the most recent articles and extracts a title and a preview of each story by using a loop. The robot then finally returns the collected values Clicking the second step in the Robot View executes the Load Page action and we see that News Magazine loads in the Browser View below. As we will see later, the Browser View makes it really intuitive to build a robot. The next tutorial will show you how to build this robot yourself. Once we have automated the process of extraction with the robot, we will upload that robot to the Management Console. The Management Console is a web-based application for managing the operational aspects of Kapow. From the robot we can create a Kapplet, which publishes the robot as an app for yourself and others to use. The final product will be a Kapplet which automatically extracts the most recent stories from News Magazine and returns them for the user to view and download. You are now ready to start building your first robot. Start the Beginner Tutorials by watching the Robot Beginner's Tutorial. Robot Beginner's Tutorial Robot beginner's tutorial video. General Introduction This is the second of four beginner tutorials which will guide you all the way through your first project with Kofax Kapow. It is advised to start with the Overview video before starting this tutorial. You are about to learn how to build a robot in Kapow's Design Studio. Please feel free to follow along on your computer. Specific Introduction In Kofax Kapow, robots are used to automate processes that can be performed in a web browser. Robots can mimic and automate any set of mouse and keyboard instructions that you would otherwise have to perform manually. 12 Kofax Kapow User's Guide In these beginner tutorials it is the goal to automate the process of extracting the most recent stories from News Magazine which is a site built specifically for these tutorials. There is a link to the site in the text associated with this video. (http://kapowsoftware.com/tutorial/news-magazine/index.html.) Under the tab Most Recent News, we find the three most recent articles on News Magazine. From these we want to extract their title and the short article preview that is given. We will design our robot to do all this. Creating a New Robot With Design Studio running, create a new robot by right-clicking the default project and choosing New > Robot.... A window opens, asking for the name of our new robot. Since this is our first robot we will call it MyFirstRobot.robot. Click next. Now direct the robot to the front page of News Magazine (http:// kapowsoftware.com/tutorial/news-magazine/index.html) and click finish. The robot editor opens and News Magazine loads. Notice that the new robot file has automatically been selected in the projects view on the left. The Robot Editor In the robot editor we have five different main views. There's the browser view which shows us the loaded page exactly like we'd expect to see it in a browser. Under the browser view, the html view shows the html of the loaded page. At the very top there's the robot view where you can see the actions performed by the robot. Actions can be anything from clicking a link to storing data in a database. The active step is highlighted in green and steps to the left of the active step have been executed. Right now the End-step, which is the small round step, is the active step and the load page step has been executed. We will get back to the meaning of the End-step later on. The step view on the right is used to configure the action performed by the active step. Since our current active step, the End-step, does not have any properties to configure, the step view contains only a description of the step. Finally the variables view specifies any variables used by the robot for input, output or for storing data during execution. The Browser View Use the browser view to navigate to the page we want to extract from. A double click in the browser view corresponds to a single click in a regular browser. Double click the Most Recent News tab to get to the stories we want to extract. Notice that a new step is created and executed in the robot view and the Most Recent News page is loaded in the browser view. Scroll down the page and see the three news articles that we want to extract from. If we single-click within the browser view, a green box appears around the HTML tag in which we have clicked. This green box marks the selected tag in the browser view. By selecting a tag and right clicking it, a menu appears which presents some actions that can be applied to the selected tag. We will use this in a moment. Looping Through Tags 13 Kofax Kapow User's Guide The next step is to make a loop which iterates through the three most recent articles. To do this we simply select any tag within the tag of the first article. Select for example the picture. Now, right-click in the selected tag and choose Loop > For Each Tag. A For Each Tag step appears in the robot view, and in the browser view a blue box appears around the first article in our loop. The For Each Tag step has arrows which can be used to iterate through the loop. Use these to iterate through the three articles and confirm that they are selected properly by the blue box. Clicking the arrows will not alter the robot in any way, but they can help us ensure that the loop has the expected iterations. Use the left-most arrow to return to the first article. The function of the end step now becomes apparent. The loop will loop over every step bounded by the For Each Tag step and the End step. Note that we cannot add any steps after the End step. Adding a Variable Now, before we can extract anything we need to add a variable to contain the text we are going to extract. As mentioned in the Introduction tutorial I have already prepared a type called post to use with this robot. Right click the white box in the variables view in the lower right corner and select Add Variable of Complex Type > post. A window opens in which we may configure the variable. Let us keep the default settings and click OK. A variable of the selected type now appears in the variables view. It contains the two attributes title and preview, which correspond to what we want to extract. Extracting Looking at the browser view again we are now ready to make the extractions. Click then right-click the title of the post marked with the blue box and choose Extract > Extract Text > title. Extracting within the blue box, called the named tag, ensures that the other iterations of the loop will extract the corresponding titles and previews from the other posts. Now do the same for the article preview. Click and right-click the preview text and choose Extract > Extract Text > preview. Notice that the extracted text is now shown in the variables view. Also notice that two extraction steps have been added and executed in the robot view. Use the arrows on the For Each Tag step, while looking at the variables view, to observe that the text is extracted properly from the other posts as well. The Return Value Step To output the collected data we now need to insert a return value step into our robot. Right-click the end step and go to the submenu "Insert Step Before" and choose Action Step. This inserts a new Action Step before the end step. Use the dropdown in the Step Action View on the right to choose the Return Value action for this step. The Step View now changes to show the properties of the Return Value action. We see that the variable we added earlier has been chosen by default. The robot is now ready to be tested in the debugger. The Debugger Switch to Debug mode by clicking the Debug button above the Robot View. The bottom part of the robot editor is now replaced with panels containing various tools to monitor the execution of the robot. The default tab in the main panel shows Input and Output. Run the robot by 14 Kofax Kapow User's Guide choosing Run from the Debug menu. The robot should now successfully execute and the output should be shown in the main panel. We have now successfully built a robot that extracts the three most recent articles from the News Magazine website. If the execution fails for some reason, you can either redo the tutorial and check all the steps or check the NewsMagazine.robot, which is a robot identical to the one we have just built. The News Magazine robot can be found in the Tutorials folder in the default project. Next step is to upload the robot to the Management Console and create a Kapplet. Move on to the Management Console tutorial or if you need help on specific topics go to help.kapowsoftware.com. Kapplet Beginner's Tutorial Kapplet Beginner's Tutorial video. This is the third of four beginner tutorials which will guide you safely through your first project with Kofax Kapow. It is advised to watch the overview video and complete the robot tutorial before following this tutorial. You are about to learn how to use the Management Console and KappZone to run robots as Kapplets. Please feel free to follow along on your computer. The Management Console is a web-based application for managing the operational aspects of Kofax Kapow. First of all, the Management Console acts as a repository for robots and types. It also includes a KappZone which enables you to create and manage Kapplets. Kapplets are robots or collections of robots that have been published as apps, easy to distribute and run. This tutorial will show you how to upload the News Magazine robot to the Management Console, and publish it as a Kapplet in the KappZone. Open Design Studio and MyFirstRobot, the robot we created in the Beginner Tutorial on Robots. Then ensure that the robot editor is in Design Mode by clicking the Design Button at the top left corner of the robot editor. To upload the robot to the Management Console, select Upload to Remote Management Console from the Tools menu. Everything should already be correctly configured so just press Upload. The robot, along with any types associated with it will now be uploaded to the Management Console. Click the link that appears to the Management Console. This should open your browser in the Repository of the Management Console. The Repository contains any robots, types and other files uploaded to the Management Console. Observe that the News Magazine robot has been listed. Similarly click the Types tab to check that the associated type has been uploaded correctly. To create a Kapplet from the robot, go to the KappZone tab at the top of the page. This opens a page with a link to the KappZone. Click it and the KappZone will open in a new window. The KappZone is where you can add, remove and edit Kapplets. To create a Kapplet click Add New Kapplet at the top of the repository. Call the new Kapplet "News Magazine" and click Create Kapplet. The Kapplet has now been created and we are taken to the configuration page of our new Kapplet. 15 Kofax Kapow User's Guide The new Kapplet is so far disabled, which is indicated by the switch at the top of the page. This means that the Kapplet is only visible to administrators. We will enable the Kapplet as soon as we are done with the configuration. There are two pages in the configuration of the Kapplet: Identity and Pages, of which Identity is selected. On the identity page it is possible to edit the name, description and icon of the Kapplet. As description I will write "Extracts the most recent news from News Magazine" and for the icon I use the News Magazine logo which I have previously saved to my computer. Going now to the pages section of the Kapplet Configuration, we need to configure what our Kapplet actually does. Kapplet functionality is structured by pages and even the simplest Kapplet has at least two pages, namely a start page and a result history page. From the Start Page the user will be able to either start the Kapplet right away or Schedule the Kapplet to start automatically at specific times. Whenever the Kapplet is started, it will run all robots added to the Start Page. The Result History page archives results returned from the executed robots. Click Add Action on the start page, then click Add New Robot, choose the robot we uploaded, click Select Robot and click OK. This adds an action to our Start Page with a button labeled "Start Kapplet". It also automatically adds a new page called Post to our Kapplet. Click the Post page on the left. This page will display a specific result from the Kapplet after it has been chosen by the user from the Result History page. Right now it displays a table containing the content of the title attribute from each result. Clicking edit on the table we can add the preview attribute to our table and click OK. The Post page will now display both the title and the preview from each of the extracted stories from News Magazine. Click to apply the changes, enable the Kapplet, and go to the menu on the right. It appears when hovering your mouse over the square in the top right corner. From the menu it is possible to go to My KappZone or just the KappZone. You can think of the KappZone as a repository which holds all the installable Kapplets. My KappZone however, contains only the Kapplets which have been installed to your account. Click on All under KappZone. We can now see that the News Magazine Kapplet has appeared in the KappZone. Hovering over it, we see that we can either install, edit or delete the Kapplet. Go ahead and install it. Immediately we are able to open the Kapplet. Going to My KappZone through the menu bar on the right, we also see that the News Magazine Kapplet has been added to My KappZone. Click on the icon to open the Kapplet. The Start Page and the Result History page of our Kapplet are now displayed. From the start page we can either schedule the Kapplet or just run it once by clicking Start Kapplet. Just go ahead and click Start Kapplet. The first result now appears in the Result History page. Click it to open the Post page showing the result and scroll to the right until the entire Post page with results appears in your browser. And so, finally our goal has been fulfilled. We have created automated process with a robot and published it as a Kapplet, easy for users to install and use. The next tutorial will teach you how to create a type in Design Studio. 16 Kofax Kapow User's Guide Type Beginner's Tutorial Type Beginner's Tutorial video. General Introduction This is the fourth of four beginner tutorials which will guide you through your first project with Kofax Kapow. It is advised to complete the other Beginner Tutorials before this tutorial. You are about to learn how to build a type in Kapow's Design Studio. Please feel free to follow along on your computer. Introduction to Types In Kofax Kapow, variables are used by robots to store data in. Robots use variables as input, output or to store temporary values during execution. These variables are categorized by types. Examples of types are text, image, PDF, number and so on. The given examples are all what we call simple types. That is, types which are predefined in Design Studio. Alternately we can create our own Complex Types. Think of a complex type as a bucket of simple types. Let me explain with an example. In these beginner tutorials we have been automating the process of extracting and storing the most recent stories from News Magazine. From these three articles we extracted the title and the short piece of text that is given. The title can be held by a variable of the simple type short text and the preview by a variable of the simple type long text. We have to design our complex type to contain each one of these. Creating a New Type With Design Studio open, create a new complex type by right-clicking the default project and choosing New>>Type... A window opens, requesting a name for the new type. Since this is our first type we will call it MyFirstType.type. Click finish and the type editor opens. Notice that the type we just created has now been highlighted in the projects view on the left. Adding Attributes The most important part of the type editor is the list of attributes associated with our new type. Attributes describe the different values that a variable of our complex type can contain. Each attribute has a name, a type, and a list of other properties associated with it. Let's start by making the title attribute. Add a new attribute by clicking the plus sign in the lower left corner of the attribute list. A new window opens in which we can configure the new attribute. Name the attribute "title" and select for it the type "Short Text". Short Text is a simple type which can contain text, no longer than one line. Click OK to add the attribute to our complex type. Likewise add an attribute named preview to contain the short article preview. This attribute should be of the type Long Text which defines a text longer than one line. What we have now made is a complex type from which we can make variables in a robot. Variables of the type MyFirstType will then be able to hold two values; title and preview. Save the type by choosing save from the file menu. 17 Kofax Kapow User's Guide Changing the robot MyFirstType.type is now exactly the same as the complex type post.type which we used in the Beginner Tutorial on robots. If we open MyFirstRobot.robot we can now switch the post variable for a variable with the type MyFirstType. Do this by right clicking the post variable in the variables view and choosing edit variable. In the Edit Variable window that opens, choose MyFirstType as the type for this variable and click OK. The last step is to save and upload the modified robot to the Management Console. It will automatically take the place of the previously uploaded version. Congratulations you have now fully completed your first project with Kofax Kapow. Advanced Tutorials This section contains tutorials on advanced topics of Kapow. Branches, Robot States, and Execution Flow This tutorial explains the concepts of Branches, Robot State, and Execution Flow. This tutorial will explain how and why to use branches in you robots. In the process it will be necessary to introduce the concept Robot States and discuss robot execution flow in general. If you have completed the beginner tutorials you will know that robot execution starts at the leftmost step and continues sequentially to the end step, where robot execution terminates. This is an example of a linear robot with no Branch Points. Before showing you the branch point we have to introduce the concept of Robot States. At every step, the robot has various elements which make up its state. The most important elements are the currently open windows and frames and current values of variables, but the state also includes cookies, authentications and so on. All of these elements make up the robot state. Back in the robot view we have now introduced a branch point. It's been inserted by selecting Add Branch from the Edit menu in Design Studio. The robot now sequentially executes each branch from top to bottom. Every time the robot reaches an end step, execution continues from the next branch. So why go through the trouble of using multiple branches, instead of having a completely linear robot? Well there are many answers to this question but the most important reason is that the robot reverts to its previous state every time execution goes back to a branch point. As we talked about before, state includes open pages, variable values, and so on, so every time the robot goes back to the branch point, it returns to the page it was on when it passed that branch point and forgets everything that happened in the branch. Let me show you an example of how to use this. 18 Kofax Kapow User's Guide I'm currently working on a robot which searches the site Momondo for travels to three different destinations but from the same departure city. The robot enters departure city into a form, then splits into three branches and enters three different destinations and clicks to search. Clicking the branch point we enter the state which the robot has when it splits into multiple branches. This is the state which the robot reverts to each time a new branch is executed. This means that the robot does not have to load the site and input the departure city three times, wasting time and CPU power. The robot simply rolls back and continues where it left off, entering a new destination into the form for each branch. You can use this technique every time one page has to be handled in multiple different ways. We can even join these three branches again since they all use the same steps for the last part of the branch. To redirect an arrow, first select it by holding Ctrl then clicking on it. Then drag the end of the arrow to the step to which you want to connect it. You can also create a new arrow by dragging from the right side of one step to the left side of another. As you may have guessed, you can make some pretty creative robot trees in this fashion, but don't panic! The execution flow is determined by one simple rule and one rule only. Once execution reaches an end step, execution will continue from the next branch of the most recently reached branch point. Let me repeat that for you. Once execution reaches an end step, execution will continue from the next branch of the most recently reached branch point. Once execution reaches an end step, execution will continue from the next branch of the most recently reached branch point. It's pretty intuitive once you get the hang of it. Okay I have a confession. There are other rules which govern execution flow and there is one exception to the rule mentioned before: For Each loops. For Each loops include the For Each Tag action, the For Each Window action, the For Each URL action, etc. If you have completed the beginner's tutorials, you have used a For Each Tag loop in your robot, and know how it works. Now we have a new way to think about For Each loops. You can think of a For Each loop step as a branch point where each iteration of the loop corresponds to a branch. In other words: Once execution reaches an end step, execution will continue from the next branch of the most recently reached branch point or from the next iteration of the most recently reached loop step, whichever comes first. Loops can be used very effectively in constellation with branches. There are also other aspects which can make robot execution flow non-linear. One of the most prominent is Error Handling. When an error occurs at a specific step, the error handling of that step decides where the robot will continue execution from. Keep this in mind. To learn more about error handling click the question mark at the top right corner of the Error Handling tab. So what if I want to keep some information from one branch to the next? Well, let's talk a little bit more about robot states, because not all elements are kept in the robot state. Global variables for example 19 Kofax Kapow User's Guide are totally linear in time throughout execution of the robot and never revert to earlier values when the robot rolls back to former states. This means that you can transfer information among branches or among iterations of a loop. You can convert any variable to a global variable by checking the checkbox Global when adding the variable to your robot. Also note that Try Steps look similar to Branch Points but they are not the same. Try Steps are only activated by error handling. Looping Basics Basic looping in your robots video. This tutorial will give you an introduction to looping within your robots. In particular we will be looking at the types of loops which can be accessed directly from the Browser View. Looping is both touched upon in the Beginner Tutorial Videos and in the video on Branches, Robot States, and Execution Flow. If you have no experience with looping, I highly recommend you to take a look at these videos before proceeding with this video. Take special notice of the way loops alter the execution flow. The most useful robots are often those which perform a large quantity of actions, simply those which get a lot done. Often this includes performing the same operations in a number of similar cases. An example is the NewsMagazine robot used in the Beginner Tutorials. This robot uses a For Each Tag step to extract text from several blog posts. The For Each Tag step is just one of many loop steps to which the same logic applies. They all somehow let you perform the same procedure in a number of related situations. The most basic loop steps are categorized as For Each Tag Loops, because they all somehow loop through tags in the current window. Let me go through these basic loops one at a time. For Each Tag The first of three loop steps we are going to discuss in this video is the loop literally called For Each Tag. For Each Tag loops over each tag of a given name directly within the found tag. The first tag in the loop has been indicated by the blue box in this screenshot of the source view. I now overlay the screenshot with lighter blue boxes to show the following iterations of the loop. For Each Tag is the loop step which I find myself using most often, simply because of its mixture of flexibility and ease of use. It is also the loop step which we used in the Beginner Tutorial Videos. Whenever I need to loop over a selection of similar tags, like the products listed on this page, the first thing I try is to right click the first element that I want and select Loop >> For Each Tag. Design Studio then sets up a For Each Tag loop which loops through tags similar to the one I right clicked. I can now iterate through the loop, using the arrows on the For Each Tag step, to view all the named tags formed by the loop. As discussed in the Beginner Tutorial Videos the blue box formed by a loop is called a named tag and is used as a point of orientation for Tag Finders of following steps, so if we insert a step which extracts the price of the first product, the following prices will be extracted likewise in subsequent iterations of the loop. This is the way all loops, considered in this video, operate. 20 Kofax Kapow User's Guide Sometimes, however, it does cause problems to insert the loop, like here on vimeo.com. I right click the first element that I want and choose the For Each Tag loop, but the resulting loop only includes the topmost listed video. We can see this by trying to go to the next iteration. This results in a window opening to tell us that we have reached the last iteration of the loop. To fix this I have to go directly to the configuration of the loop step where, in this case, I need to remove the specification of class to loop over. Design Studio guessed that we only wanted tags with the class top but really we want to loop over every listed item, independent of class. I delete the class specification and the loop now works as expected. To use For Each Tag effectively you should study the different ways to configure the For Each Tag step. For Each Table Row/Column For the following loops it should be mentioned that they are often interchangeable and a given situation may be handled in any number of ways. I will try to teach you the basic principles of each loop type so you can be intelligent about which type you choose, but there is no single correct way of doing things. The two next types of loop steps we will look at are both derivatives of the For Each Tag step, but they have more specific uses. They are called For Each Table Row and For Each Table Column and they respectively loop through rows and columns of a table. As with the For Each Tag step they have been conveniently implemented in the right click menu. In this screenshot the first row is shown with the named tag marked 1 and the first column is the named tag marked 2. As you can see, combining the two types of loops will let you loop over every element in a table. To insert a loop over table rows or columns, right click on any table element and select the appropriate action from the Loop submenu. You may either choose to include or exclude the first row or column. I have now inserted a loop which loops through each column of the newest Ikea furniture. Notice that the name of the loop step is For Each Tag. Instead of having a unique step for looping through tables, the For Each Tag step has just been automatically configured to loop through columns in a table. For Each Tag Path The next type of For Each Tag loop is called For Each Tag Path. It is very similar to For Each Tag, which we just discussed. The difference between the two is that For Each Tag Path loops over tags that are at any level inside the found tag whereas For Each Tag only loops over tags that are directly inside the found tag. Sometimes the tags you want to loop over are not all directly inside one parent tag, or possibly the tags you want to loop over are all on different levels then you will need to use the For Each Tag Path loop. Notice in this example how the div tags looped over are all within a td and a tr tag and are therefore not directly within the found tag. The easiest way to determine whether to use For Each Tag or For Each Tag Path is to look at the page structure in the Source View. For Tags with Class 21 Kofax Kapow User's Guide Just like For Each Table Row and Column were derivatives of the For Each Tag loop as is the For Tags with Class a derivative of the For Each Tag Path loop. As an example of the For Each Tag Path loop let me show you how to use this derived version. Okay, now let us delete the table column loop we set up and take a look at the For Tags with Class. This loop iterates over all tags with the same value of their class attribute, which is often the case for tags with similar content. This time we have to be a bit more specific which tag we select before right clicking and we also have to keep an eye on the source view. Usually using this loop goes something like this: As we click on the tags containing each product we look in the source view and notice that they all have the same class, namely productContainer. Once we realize this we can simply right click on one of the tags and choose Loops >> For Tags with Class >> productContainer. We then iterate through the loop to check that the named tags match our expectations. Again notice that the inserted step is not called For Tags with Class but rather For Each Tag Path, which has simply been configured to perform the specific task. For Each URL The last loop we will take a look at is the For Each URL action, which is in a category by itself. For Each URL simply loops through each URL inside the found tag. It is often useful if you need to extract or click on every link in a specific area of a page, regardless of the context of the link. For Each URL is most easily inserted by selecting the tag containing the links you would like to loop over, then right clicking the selection and choosing Loop >> For Each URL. I have now set up a loop which iterates through each URL in this article. It by default skips duplicate URLs. Let's leave the For Each URL action at that. Just note that For Each URL has a number of configuration possibilities which can be changed in the step view. Finally I have two notes that will help you when using loops. Note 1 Just to spell out what I said in the video on Branches, Robot States, and Execution Flow: A For Each loop step, like any of those in this video, executes every subsequent step in the robot view for every iteration of the loop, so if you want your robot to continue execution beyond the loop, then you will have to insert a separate branch before the loop step. This branch will then be executed after the loop has finished. Note 2 It is often nice to be able to break a loop or skip an iteration based on certain conditions. If we for example reach an iteration where one of the steps within the loop cannot be performed, it would often be logical to skip this iteration altogether. As mentioned in the video on Branches, Robot States, and Execution Flow this can be done by adjusting the Error Handling of the step that fails. In the Step Error Handling View you can choose Next Iteration or Break Loop if an error occurs at this step. At default this option is set to Skip Following Steps which corresponds to letting the robot hit an end step at its current execution position. In other words, if an error occurs at this step, the robot will go back and execute the next branch of the most recently reached branch point or the next iteration of the most 22 Kofax Kapow User's Guide recently reached loop step, however it will also cause an API Exception and Log an Error as indicated by the check boxes. These were just the basics of looping. To learn more check out the Loops in Forms and Repeat-Next Loop videos. Also feel free to go to help.kapowsoftware.com to read about loops in greater detail. Loops in Forms A video demo of the loop steps which apply to forms. This tutorial will give insight into types of loops which are useful when working with forms. It is a direct continuation of the Looping Basics video, and will extend on the knowledge obtained there. Many of the loops discussed in this video will have a form similar to the For Each Tag loops from the Looping Basics video but the form loops are all different from the For Each Tag loops in that they don't assign Named Tags for each iteration, instead they perform some other action for each element they work on. For Each Loops in Forms There are two For Each Loops and one other loop specifically designed to be used with forms. They are called For Each Radio Button, For Each Option and Loop Field Values. I will be demonstrating each of the Form Loops on this library search page. For Each Radio Button does what you would expect; it selects one radio button in a group for each iteration. The easiest way to insert the step is to right click a radio button and select For Each Radio Button in the Loop submenu. Iterating through the loop we see that each radio button is selected sequentially. The next form specific loop is called For Each Option. It loops through options in a drop down list. Again it is easily selected from the right click menu. Once selected; a window appears asking whether you would like to skip any of the options of the drop down. This is useful for skipping any options that are not appropriate for what you want to do. In this case I will loop through all the options. Each option is now selected when clicking the arrows on the loop step. The last loop I want to discuss in relation to forms is called Loop Field Values . This is not a For Each loop, which means that it is used slightly differently from the other loops I have described. It can be used on any text input field to loop through listed values that are then inserted into the field. One value for each iteration of the loop. Again we simply right click and select the Loop Field Values option. We can now choose which types of values to loop through. We can either choose one of the predefined options or we can compile our own list of values that we would like the robot to input. Since we are searching a library I will compile my own list consisting of Hemingway, Shakespeare, and Poe. Iterating through the loop we now observe that the stated authors are entered in to the text input field. That concludes this video on Loops in Forms. To learn more go to help.kapowsoftware.com. Repeat-Next Click to open a video demo introducing the Repeat-Next loop. This tutorial will give insight into an advanced but very useful loop called a Repeat-Next loop, which works particularly well for looping through pages where each page leads to the next. 23 Kofax Kapow User's Guide We will build upon the skills learned in the Looping Basics video, so make sure to watch that first. Repeat-Next This loop type is the oddball in the looping family, or maybe you could argue that it is not even part of the family because the way this loop is designed is totally different from all the other loop steps in Design Studio. First of all the Repeat Next loop consists of two individual steps which need to be used collaboratively to have any kind of effect. The concept is actually pretty simple: you place a Repeat step followed at some point by a Next step. When execution reaches the Next step, execution will revert back to the Repeat step and proceed execution from there, marking one iteration of the loop. If a Next step is not reached after the Repeat step then the loop will terminate. The catch here, and also what makes the Repeat-Next loop so genius, is that while most of the robot state is reverted at the beginning of each iteration, the page reached at the Next step is actually transferred to the next iteration of the loop. So unlike the other loops we have looked at where the entire robot state is reverted at the beginning of each iteration, we can handle a different page in each iteration of the RepeatNext loop. Demo A typical example of using the Repeat Next loop is when looping through multiple pages of search results. We insert a Repeat step … followed by a step which clicks to get to the next page … and then a Next step … It's as simple as that to loop through all the pages. We can now use the arrows on the Repeat step to iterate through the loop and observe that a new page is loaded for every iteration. Of course we still need a way to terminate the loop. This can be done by setting the error handling of the Click step to "break loop". If the click step does not find a link to the next page, we assume that we must have reached the last page and the loop breaks. If we then want to perform some steps on each page, we can add a branch step after the Repeat step … and add a new top branch. In this new branch we can add loops and other steps without them being influenced by the Click and Next actions in the other branch. I can for example add a loop over all the search results on each page … extract information from each result … and return that collected information. In total I get a robot which extracts every result from every page of the search. We can get an idea about the execution flow by single-stepping through the robot in Debug Mode. I have sped up the recording so you can clearly see how all search results are extracted, one page at a time. Remember the form in which the Repeat-Next loop is used here. A top branch which executes the steps you want to perform on each page and a lower branch which loads the next page and calls the next iteration of the loop. This constellation is useful and very common when using this type of loop. Note that setting the error handling of a step to "Next Iteration" does not work with Repeat-Next loops. In order to proceed to the next iteration, a Next step must have been executed. Try Step Click to see a video explaining how to use the try step to set up conditions and error handling in your robots. 24 Kofax Kapow User's Guide This video will demonstrate how to use try steps when building robots in Design Studio. Applications of the try step are crucial for building robust robots acting on a dynamic web environment where changes in structure are commonplace. Use cases include trying multiple different strategies for interacting with or extracting from a website, and setting up conditional statements in a robot. We will go through two examples in this video, demonstrating the two use cases, starting with the latter. As a first example we'll be using ExtractFromTable.robot, which is a basic robot that extracts data from a table on the News Magazine website. It can be found under Robots in the Examples folder of the default project. I open the robot and hide the projects view. Once open, you will notice that the robot has only one path to follow in the robot view until it reaches this diamond shaped step called a try step. What a try step does is set up multiple alternatives for the robot to try. A try step can have any number of alternatives which are shown as dashed arrows emerging from the right side of the try step. If an alternative succeeds the robot continues as usual, ignoring any other alternatives from that try step. If an alternative does not succeed, however, the robot goes back to the try step and tries the next alternative. To use try steps it is therefore important to understand what I means for an alternative to "succeed" or "not succeed"? Let me try to demonstrate with this example robot. ExtractFromTable is a simple robot that extracts person information from a table by using a loop. Clicking the "for each tag" step and scrolling down reveals the table which the robot extracts from. Inside the loop, which loops through the rows in the table, the four pieces of information for each person are assigned to different attributes of the person variable. The try step is used to assign either true or false to the isMale attribute, based on the value of the last column in the table, called sex. The column can have either one of two values, "Male" or "Female" and we want to assign true or false to the isMale attribute respectively. To test the value in the sex column, a Test Tag step has been inserted in the first alternative. Test Tag is a step which can conduct an action based on whether the found tag matches a specified pattern. Clicking the Test Tag step we see that the found tag is the cell in the sex column of the row from which we are extracting in the current iteration, and the pattern to match is simply set to "male" and only matches against text, ignoring case. If the pattern matches, the step then does what is specified in the Error Handling Tab. Under Error Handling the step has been set to "Try Next Alternative", which is also indicated by the small icon in the upper left corner of the step. This means that if the sex is male, and the pattern is thereby matched, the Test Tag step will do what is specified under Error Handling and try the next alternative. The second alternative therefore, has a step that assigns "true" to the isMale attribute. If the pattern is however not matched, the Test Tag step does nothing and execution proceeds normally to the next step, which assigns false to the isMale attribute. In other words, an alternative can be said to succeed if no errors which specify to try next alternative are encountered, and can be said to not succeed if such an error is encountered. As soon as one alternative succeeds, all other alternatives are skipped. In the example I just went through, the error was generated by the Test Tag step, but usually errors are caused by steps which are unable to perform their actions. This is most often because steps cannot find a certain tag or loading of certain content times out, but it could be anything that stops a step from performing its action. Executing the example robot in debug mode you will see that the correct values are assigned to the isMale attribute. True values are indicated by check marks and false values are indicated by the absence thereof. 25 Kofax Kapow User's Guide To give you an even better understanding of how the try step works, let me show a more complex, yet common, use case. In Design Studio I have this robot which needs to log into a site and extract some data, which is only available upon login. The page used by the robot is of course the familiar News Magazine page. The robot is called Login and is available from the examples folder. If you have closed the Projects View, it can be reopened from the Window menu. Look at the robot view to get an overview: In the login process, the robot uses save and restore session in such a manner that the robot logs into the site and saves the session on the first run. Consecutive runs will then load the saved session instead of performing the login sequence again. At some point the session expires and the robot will have to log in and save a new session. Most robot executions will thus be able to restore a session and be able to skip the entire login procedure. So… we have three different scenarios to test for: either it is the first time the robot executes and there is no saved session, or there is a saved session which is already logged in, or there is a saved session but it has expired. Looking at the robot view, there are two alternative paths for the robot to take, depending on whether it is able to use the stored session or not. To show each case, let's try simulating them by clicking through the robot. The first time the robot is executed there is no saved session. The robot will first try the topmost alternative from the try step, but if I try executing the Restore Session step it will fail, since there is no session to restore. When this step fails, it will cause an error, triggering error handling which is set to "Try Next Alternative". The robot will therefore have to execute the second alternative, which follows this lowermost branch, loading News Magazine, clicking to get to the login page, entering username and password, clicking to login, saving the session for future executions of the robot, clicking again to get to the data we want to extract, and finally extracting the specific text. We have now simulated the first run of the robot. Going back to the first step in the robot, let's simulate the second run. In all executions following the first, the session should restore successfully, so following the topmost alternative the session is restored, the robot clicks to open the page that we want to extract data from, and finally the data is extracted - simple. We now want to replicate what happens if the robot is to execute after the login session has expired. To do this, open Logout.robot from the examples folder. The logout robot restores the saved session and clicks logout on the News Magazine page. In this way we are emulating expiration of the login session, rather than actually waiting the 10 minutes it takes for the session to timeout and expire on this specific page. Click the end step in the logout robot to execute all steps. Going back to the login robot click the step named "Click site data" to make it the active step and click refresh in the toolbar to re-execute all steps up until the active step. After all the steps have executed again, notice that the page shown in the browser view looks as though we are still logged into News Magazine. This is because the saved session also contains the entire robot state, including the html of the page we were on when the session was saved. Restoring the session therefore does not properly refresh the page to show us its current state. Clicking on the extract step to execute the click step will however refresh the browser to show us that we are no longer logged in. The extract step therefore fails to extract the data we want, causes an error and tries the next alternative from the try step, which logs in and saves a new session for future use. Thus the login robot handles all three cases and thereby ensures that the robot always logs in before extracting the data which is otherwise not available. 26 Kofax Kapow User's Guide To summarize on what we have looked at in this video, from a try step spring multiple alternative routes for the robot to take. An alternative succeeds unless it has a step that causes an error and specifies to "Try Next Alternative" in the Error Handling Tab. Only the first alternative that succeeds is executed, the others are completely ignored. If an alternative does not succeed, the robot goes back and tries the next alternative. Here are a couple of tips when using try steps: As we saw in the first example, any condition steps, meaning steps with "Test" in their names, work well together with the try step. If no alternatives of a try step succeed then the try step itself generates an error. Specify what action to take in the error handling tab of the try step itself. You can give a try step a name by double clicking below it in the robot view. When specifying to "try next alternative" under error handling for any step, you can also choose which try step to go back to based on their names. This means you can have nested try steps allowing for complex robot structures. That concludes this video on the topic of try steps. Excel Click to watch a video explaining how to read data from Excel documents. Starting from Kapow version 9.2, robots have a whole new way of handling Excel documents. Instead of being converted into html-pages, Excel is now shown as a spreadsheet directly in the Page View in Design Studio. In this video we will give an overview of the features of the Spreadsheet View and take you through the process of building a robot which extracts from an Excel document. Feel free to follow along on your own machine. An example of a robot which loads an Excel document can be found in the examples folder of the default project, and has the name excel.robot. Go ahead and open the robot. Excel documents can be loaded in the same way as regular pages are loaded, either from a URL or from a file on your machine or server. In this particular robot it is loaded by clicking a link, which is performed by the step named Click PersonData.xlsx. Click the Loop Rows step to view the document. The spreadsheet now appears in the page view, with rows and columns named and arranged like you are used to it from Excel. This particular document contains tables with person data. There are two different sheets with 100 entries in each. It is possible to switch between the two sheets in the lower left corner of the Page View. It is also possible to see the document information by clicking the leftmost tab. In this video however, I will focus on Sheet1. From the drop down menu in the lower right corner of the page view it is possible to look at the unformatted values of the document or the raw formulas. Changing this view will also affect the extracted values when inserting a new step, so I am going to change it back to showing formatted values. 27 Kofax Kapow User's Guide A cell is selected by clicking on it and it is even possible to select a range of cells by clicking and dragging, or clicking on a row or column name to select the entire row or column. The upper left corner selects the entire spreadsheet. I will now delete the loop step and all the extract steps from the robot and demonstrate how easy it is to extract from an Excel document. I drag to select the steps, then hit the delete button on my keyboard. To the new Spreadsheet View belongs a whole family of new step actions, which enable you to loop, extract, and test cells. Just like in the browser view these functions are available from the right click menu as I will show you in a moment. First we want to insert a loop step which loops through all the rows of the table. First I click the upper left corner of the Spreadsheet View to select the entire spreadsheet. Then I right click inside the selected area - meaning anywhere inside the Spreadsheet View- and select Loop>>Loop Table Row>>Exclude First Row. I am excluding the first row since we are not interested in the header values. The Loop in Excel step now sets the first row to loop over as the Named Range. Clicking the arrows on the loop step will show how the other rows are selected. It is now possible to extract from the Named Range and, because of the loop, corresponding values will then be extracted from all the other rows. To extract first the ID, I right click the ID value and select Extract > Extract Number > ID. I click OK in the wizard that appears, which is already properly configured. Likewise I can now extract the first name as a text into the name variable, the age as a number into the age variable and the gender as a Boolean into the isMale variable. The gender value is either true for male or false for female. To show that the entire table can now be extracted, I switch to debug mode by clicking the icon in the upper left corner and clicking run in the toolbar. All 100 values from the Excel document now appear in the list of results. Writing to Excel Click to watch a video tutorial on how to write to an Excel document. This tutorial video will show you how to make a robot that automatically creates an excel document based on data extracted from a website. Please feel free to pause the video along the way so you can follow the steps on your own computer. With Design Studio open, we go into the Examples folder in the default project. Under Robots, we find and open the example robot called TableExtract.robot. This is a robot which extracts person data from a table on the website News Magazine. We can see the table by clicking on the loop step. Before moving on, make sure you familiarize yourself with this robot. We will modify the robot to write the extracted data into excel. We first need to clean up the robot a bit. We start by deleting the two first steps, which are only there to display help about the original robot. Then we add a new type to the robot by right clicking in the Variables view and choosing Add Variable of Complex Type > PersonListExcel. This type contains an Excel attribute, which the robot will write to. Let's call the variable personList and mark it as Global. Having a global variable ensures that we can add content to the spread sheet over multiple iterations of a loop in our robot. We click OK. 28 Kofax Kapow User's Guide To modify our Excel variable, we must open it in the page view. We right click personList and choose Insert Step >> Open Variable >> personList.list. This inserts and executes an Open Variable step in the Robot View and, although the sheet empty, we can now see the content of our personList variable. Next, we add a header for each column in the table we are extracting from. To insert content into Excel, we right click a cell and choose Modify >> Set >> Text. This opens a dialog where we can specify the value to insert. We write "id" and click OK. Then we repeat this process for the next three cells which we call "name", "age" and "isMale". It's now time to make the robot insert the extracted data into this spread sheet. We click on the end step of the robot. An error might occur, stating that the Test Tag step stopped execution. This is an intended error. To get around it, use the lower alternative from the Try Step. We switch to the Excel window by right clicking it and choosing Set as Current Window. This inserts and executes a Set Current Window step in the Robot View. Before we can insert new content into the Excel variable, we must extend the sheet with a new row. We select the entire sheet by clicking the upper left corner of the Excel View. Then we can right click on the sheet and choose Modify > Insert > Rows > Last. A dialog opens, asking how many rows to insert. We only need to insert one row for each iteration so we press OK. This will insert a new row as the last row in our sheet and mark it as a named range. We can now select the named range by clicking the second row on the very left side of the Excel View. We then right click the selected row and chooseModify > Set > Content of Row > person to insert the data extracted from News Magazine. Stepping thought the iterations of the loop, we can see how multiple rows of content are added to our Excel variable. So now we have a robot which creates and writes to an Excel variable. To get full usage of our robot, we need to make it save the Excel variable as a document on the hard drive. To this extent, we insert a branch just before our loop step. The lower branch will be executed after the loop has finished all iterations. Stepping into the lower branch, we add a new action step and select for it the Write File action, which can be found under the File System category. To configure the Write File step, we need to give a file path. I will simply place the file at the root of my C: \ drive and call it simpleExcel.xlsx. You may choose to place it wherever you want. The file content should be our personList variable. Running the robot in Debug Mode will now write the Excel variable as a document at the specified location. If we open the file, we see that our spread sheet looks as expected. This was a very simple example of how to write to excel in a robot. For a more advanced example take a look at ExcelAdvanced.robot in the examples folder or read about the capabilities of writing to excel in the documentation in Kapow help. Data Conversion Click to watch a video describing how to perform data conversion in robots. 29 Kofax Kapow User's Guide Sometimes when designing a robot we need to convert some text or numbers using simple or complex conversion rules. The question is: how do we do this easily in Design Studio? In this video we will be talking about this field. It may not look like much but it is actually a very powerful tool when building robots and it pops up everywhere in the Design Studio interface. The field is called a Data Converter List, and for now let's regard it as a black box. It may take an input in the form of a value from, for example, a variable or extracted text; but input is not a strict necessity. The input, if present, is given by the placement of the Data Converter List and is always stated somewhere above it. The Data Converter List has exactly one output, which is either stored in a variable or handled otherwise below. In short, a Data Converter List is used to manipulate text and numbers within robots, but although it is possible to perform both text and number manipulation with the Data Converter List, input and output is strictly speaking always interpreted as text. Some use examples include extracting the name from an email address,.. ..multiplying a number by two,.. or adding three days and two hours to a given date. Let's take a look at the interior. As the name Data Converter List implies, the field contains a list which you can add a variety of different converters to by clicking the plus icon. These converters all manipulate the input in some way. Here is an example where we have the Kapow Software website URL as the input. We start with no converters in the converter list so the output is the same as the input. By adding the Extract converter we can extract the middle part of the URL. Don't worry about how the converters work for now. We will get back to that. Then we can add a converter which converts the text to upper case. The two converters are then chained together in series so the output of the first flows into the input of the second. The order in which the converters are listed represents the order in which they are performed. The order can be changed by clicking the arrows. In this case it makes no difference to the end result. Converters can be deleted by clicking the minus. Clicking the pen and paper icon opens a window used to configure the converter. This window also opens when the converter is first added to the list. The Converter configuration window always contains the two important fields Test Input and Test Output which, as the names imply, demonstrate the input and output from the given converter. Learn how to use the different converters by clicking the question mark in the configuration window. Clicking it opens the documentation. Here you can also read about Expressions and Patterns. Knowing how to use these is an invaluable skill when working with data converters. 30 Kofax Kapow User's Guide For example if you need to use mathematical operations to manipulate a number, expressions are needed. I can multiply the input by two by using the converter called evaluate expression. Let's take a closer look at the expression used to perform this operation. It converts the input to a number and then multiplies it by two. Remember that all input and output from converters is text so we need to convert the input to a number before applying any mathematical operations to it. Here are some examples of the use of data converter lists in Design Studio. The data converter list you will find the most useful is probably the one located in the Extract step action. It allows you to convert any extracted text before storing it in a variable. Let me show you an example of how to use this. I'm in the process of creating a robot which extracts job offerings from LinkedIn. For now it simply loads the page and loops through the tags which contain job descriptions. I would like to extract the location of each job but it's stated in the same tag as two other pieces of information, namely company name and date. The three pieces of information are only separated by hyphens. I start by extracting the entire text into a variable called location. I then select the Extraction step in the robot view and add an extract converter to the data converter list of the extraction step. The extract converter uses a pattern to decide which part of the input should be extracted and used as output. In this case we want to extract everything within the first and the second hyphen. After finishing the pattern we can see that the Test Output field reflects the text we wanted to extract. Selecting the end step and iterating through the loop we can see that the location has successfully been extracted from each of the job descriptions. As you can see, converters are quickly implemented and extremely effective. Lastly I'll give you a couple of tips on the use of data converters. Tip 1: Many fields in Design Studio can be changed to data converter lists. This may be done by choosing Converters from the Value Selector located at the right side of the field. Tip 2: One of the most useful converters is the Replace Pattern converter. It combines expressions and patterns to provide powerful text manipulation. Tip 3: In addition to the restricted one input of data converter lists, additional variables can be fetched by using expressions. This allows us to combine data from several variables. Thank you for watching this short introduction to data converters. For more information on the available converters, see Data Converters in the Reference section. Patterns Click to watch a video on patterns and their use in Design Studio. The first half is a lecture like presentation of the syntax, the second half looks closer at some use-case examples in Design Studio. Answers to the problems given in the video can be found at the bottom of the page. Hello. This video will take a closer look at regular expressions, called patterns in Kapow. The first half of the video will be a lecture-like presentation of the syntax including wild cards, sets, subpatterns, repetition operators, alternate subpatterns, and subpattern references. The second half will go through 31 Kofax Kapow User's Guide three examples in Design Studio using patterns to create conditions and tag finders and to perform data conversion. If you're already familiar with regular expressions you might want to skip directly to the examples. As mentioned earlier, regular expressions are called patterns in Kapow, and will be referred to as patterns for the remainder of this video. The Wild Card A pattern is a way to put a string of characters into more general terms by using symbols to represent strings of characters. You might be familiar with the concept from doing searches on your computer where it is sometimes possible to use wild card symbols to represent any character. Doing a search for 'ca*' (the asterisk being the wild card in this example) might return both "cap", "car", "can", and so on. Patterns embrace this same concept while expanding to a much more extensive syntax, which will be presented here. Kapow uses the Perl5 syntax for its patterns. In this syntax the wild card character is symbolized with '.' (a dot or period) which corresponds to any single character including all symbols, whitespaces, and any other special characters you could think of. This correspondence is called matching so the pattern 'ca.' is for example said to match "cap", "car", "can", or any other string of "c" followed by "a" followed by any single character. Similarly the pattern '.a.' matches "nap", "tan", "sad", or any other string of three characters with an "a" as the middle character. It does however not match "an" since each '.' in the pattern has to match up against exactly one character. Similarly it does not match "cans" since the pattern has to match the entire string, not just part of it. We can test whether a pattern matches a given string directly in Design Studio by using the Pattern Editor. The Pattern Editor can for example be found by inserting a Test Tag step into a robot and clicking "Edit…" below the pattern field in the step action view. The Pattern Editor has three sections. At the top it is possible to type in a pattern, which is then matched to the string typed into the Input field on the left. Clicking the Test button or using the shortcut Ctrl+Enter will tell you whether the pattern matches the input. Try typing '.a.' in the Pattern field and "can" in the Input field. Then use the shortcut Ctrl+Enter. The Output field will now display "The pattern matches the input." We can ignore the rest of the output for now. If we on the other hand type just "an" in the input field and press Ctrl+Enter we receive the message that "The pattern does not match the input." As I go over more of the pattern syntax, try to experiment with the Pattern Editor to test your understanding of the material. Although not stated explicitly we are now able to match in two different ways, either we can match character to character (ie. The pattern 'a' matches the string "a") or we can use the wild card symbol '.' to match any character. Additional direct character matching includes the ones listed in the table here. Pattern Matches the string '\n' A line break character. '\r' A carriage return character. '\t' A tab character. '\.' "." '\\' "\" Any other symbol used by the pattern syntax can also be explicitly matched by preceding it by a backslash '\'. 32 Kofax Kapow User's Guide Sets Next step is to match a semi known character. By semi known I mean that we only want to match the character with one character in a set of characters. A set of characters is stated in a pattern by using '[]' (brackets). An example is '[abc]' (the set of a, b, c) which will match to either "a", "b" or "c" but will not match to any other characters than these three. If you wish to include a range of characters to a set it can be done using a '-' (dash or hyphen). '[abc]' can therefore be written as '[a-c]' (the set of characters: a through c). Using words '[a-c]' means match any character in the range from "a" to "c". The two ways of defining sets can be combined to get something like '[a-dkx-z]' (the set of a through d, k, and x through z) which is similar to writing '[abcdkxyz]' (out all those characters in a set) or saying match any character which is either in the range "a" to "d", is "k", or is in the range "x" to "z". It is also possible to define sets negatively by using '[^]' (a caret at the beginning of the set). An example is '[^a-c]' (the negative set of a through c) which will match any one character excluding "a", "b", and "c". In the Pattern Editor, try using sets to match (1) any digit (2) any whitespace character (3) anything that is not a digit. You can pause the video if you want to take a moment to think about these problems before seeing the answers (the answers are all given at the bottom of this page). There are certain shortcuts which can be used for sets that are often used. Here is a table showing some of the most important ones. Shorthand form Set '\d' '[0-9]' (Any digit) '\D' '[^0-9]' (Any non-digit) '\s' '[ \n\r\t]' (Any whitespace character) '\S' '[^ \n\r\t]' (Any non-whitespace character) '\w' '[a-zA-Z0-9_]' (Any word character) '\W' '[^a-zA-Z0-9_]' (Any non-word character) Note The shorthand form can also be used inside sets. For example '[\d\w]' includes all digit and whitespace characters. Subpatterns Next we will need to talk about subpatterns within patterns. Terms we have talked about so far such as a character 'a', a set '[abc]', an escaped character '\d' or the wildcard '.' can each be seen as a subpattern. Alternatively we can create our own subpatterns by grouping together other subpatterns using '()'. We could for example create a subpattern from '[ctb]an' by writing '([ctb]an)'. It is important to recognize these since I will now be introducing some operators which work on the entire subpattern they follow. Repetition Operators Operators in patterns allow us to match repetitions of a subpattern by following them with one of the operators given in the table. Repetition Operator Meaning '{m,n}' where n ≥ m Matches between m and n repetitions (inclusively) of the preceding subpattern. 33 Kofax Kapow User's Guide Repetition Operator Meaning '{m,}' Matches m or more repetitions of the preceding subpattern. For example, the pattern 'a{1,}' would match the string "a", "aa", "aaa", or any number of repetitions of 'a'. The pattern '([bn]a){3,3}' would match 'banana', 'babana', 'nabana', or any other string of either "b" or "n" followed by "a" repeated three times. Try it out for yourself. As for the sets, there are also shorthand versions of the most useful repetition operators as shown in this table. Shorthand operator Corresponds to '{m}' '{m,m}' '?' '{0,1}' '*' '{0,}' '+' '{1,}' Try using what we have learned so far to match (4) anything (5) either "color" spelled without a "u" or "colour" spelled with a "u" (6) any four digit number. The answers follow (at the bottom of the page). One of the often used patterns is '.*' which matches anything: any string even if it's empty. Now try extending this and find patterns that match (7) any text containing at least one digit (8) any text containing just one digit. Here is a list of the syntax you may need (video only). The syntax used in the answers is very useful when matching specific subpatterns within a string. Alternative Subpatterns We discussed how to match alternative characters earlier, but what about matching alternative subpatterns? If we have N subpatterns 'p1' through 'pN' , we can match any one of these subpatterns using '(p1|p2|…|pN)' (parentheses and vertical bars as shown here). The pattern given here '(abc|a{5}|\d)' would for example match with either "abc", "aaaaa" or any number. Try using alternative subpatterns to make a pattern that matches (9) a string which does not contain just one digit. Here, again, is the syntax you might need. And here is the answer: (page bottom) There is no not operator in the syntax, instead the answer uses two alternatives. The first alternative matches a string with no digits, the second matches any string containing at least two digits. Subpattern References The last major part of the syntax to cover is subpattern references. Any substring, "s1" through "sN", matched by a parenthesized subpattern, '(p1)' through '(pN)' in any one pattern, can be referenced to by using '\1' through '\N' where each subpattern is numbered in order from left to right as they are stated in the pattern. Matching '([chm])(at)' to "cat" for example, we could use the reference '\1' to refer to "c" and '\2' to refer to "at". The entire pattern can always be matched by '\0'. Notice here that we are referring to the string matched by that subpattern rather than the subpattern itself. A reference to the subpattern '(abc)' would of course yield 'abc' whereas a reference to the subpattern '(\d)' would only match whatever digit was matched by the original subpattern. As an example consider matching a string containing a quote by using the pattern '.*(['"]).*\1.*' (anything followed by a single or double quote followed by anything followed by a reference followed by anything). This may look confusing but the only thing you really need to notice is that the reference will match the same type of quote which was matched by the subpattern. In other words, this pattern would match both 34 Kofax Kapow User's Guide the string He said "hello" with double quotes and He said 'hello' with single quotes. I have purposefully not quoted the two strings here to avoid confusion. As I will show you later in Design Studio, subpatterns can also be referred to in certain expressions outside of patterns. This is useful when extracting certain parts of a matched string. Taking our quotes example we could add parentheses around the subpattern enclosed by quotes '.*(['"])(.*)\1.*'. Now we are able to extract the quote in Design Studio. Here is another problem. Try using subpattern references to match (10) four of the same digit (11) a string where at least two characters are the same. … The answers are given (at the bottom of the page) here. Fewer Repetitions When using subpattern references it is handy to know the following. By default, the repetition pattern operators (*, +, {...}) will match as many repetitions of the preceding pattern as possible. You can put a "?" after a repetition operator to instead make it match as few repetitions as possible. (12) Try matching a subpattern to the first occurrence of a digit in a string. … the answer is given (at the bottom of the page) here. Removing '?' would result in matching the subpattern to the last occurrence of a digit in the string. Using Patterns in Design Studio Now that we have learned the syntax of patterns it is time to look at the various use-cases in Design Studio. Conditions Creating conditions is the first way of using patterns intelligently in robots. The Test Tag step action is particularly relevant in this context so let's go over a common use case. I here have a robot which extracts from LinkedIn, all engineering jobs they have listed for Denmark. The robot uses a loop to extract the URL, title, and company name from each job and return them to the user. But let's say I only want to extract from jobs which contain the words "Copenhagen" and "software", indicating that they are probably looking for software engineers in Copenhagen. First, I insert a new step after the For Each step and assign to it the Test Tag action by clicking on the new step to select it and choose Test Tag from the drop down in the step action view. I ensure that the tag finder finds the entire job post of the current iteration of the loop. Then I iterate through the loop until I find a job offering which matches the criteria I am about to set. This makes it easier to test that the pattern I write will actually work. Going to the action tab in the step view, I first choose to match only against text (not the entire HTML), then press edit on the pattern. I am now in the Pattern Editor and I can type a pattern to be matched. Since I do not know the order in which the two words "software" and "Copenhagen" might occur, I need to make two alternative subpatterns. In the first alternative I can have Copenhagen followed by anything followed by software. In the second alternative I write the same but in reverse order. Finally I add "any text" before and after the alternatives and press Ctrl+Enter to test whether the pattern matches. It matches! I close the Pattern Editor and set the Test Tag step to Skip the Following Steps if the Pattern Does Not Match the Found Tag. This way the job post will be skipped if it does not contain the two words specified. I now go ahead and run the robot in Debug Mode. As expected only few results are extracted and they should all contain the words Software and Copenhagen. Tag Finders Patterns can also be used in tag finders. This can be very useful if you know the structure of the information you are looking for but you do not know where on the page it is located. This robot for 35 Kofax Kapow User's Guide example goes to multiple different sites to extract the price of a certain pair of headphones. Since we cannot know where on the page to find the price, patterns play a crucial part in determining exactly this. Let me show you how to set up the extraction step. I'll delete the one I already have, insert a new step and choose for it the Extract action. To configure the step I start by inserting a number converter which extracts the number from any text I might extract. Then I choose to extract into the price attribute of the variable I have made for this robot. Going to the Finders tab in the step view, I click plus to add a Tag Finder. I locate the price on the page. I can see that it is secluded in its own tag, with nothing else in that tag. This is typical so we will let our pattern match this case. In the Finders View there is a field called Tag Pattern. Immediately we can write the pattern '\$[\d\.]+' (dollar sign followed by one or more digits or dots). The pattern is designed to match any tag containing only a dollar sign followed by a decimal number. I click the magnifying glass in the upper right corner of the page view, which shows me what the Tag Finder finds. Unfortunately it finds the cart balance instead of the headphone price. The cart balance will always be $0 for these kinds of sites, so to avoid this mistake, I will make sure that the first digit in my tag is not a zero. Fortunately, the steep price of headphones ensures that the price will never start with a zero. Rewriting the pattern I get '\$[1-9] [\d\.]+' (dollar sign followed by a digit which is not a zero followed by one or more digits or dots) which finds the correct price on the page when I click the magnifying glass. Before testing the robot I go to the error handling tab of the Extract step and choose to Ignore and Continue on error. If the Tag Finder fails to find the price on the page it should just return the default value of the price attribute which is set to -1. This gives me a clear indication that the robot was not able to find the price. Going to Debug Mode and Looking at the results from an earlier execution of this robot, we see that many of the prices are extracted correctly. The method is of course flawed but it can be surprisingly effective at times. Data Conversion The final use for patterns is to convert data from one form into another… For this we can either use one of the data converter lists embedded in a step or use the dedicated Convert Variables step. In this very simple example, I am extracting the author and date from a blog post. Unfortunately, the two pieces of information are contained by the same string of text and are therefore extracted collectively by the extract step. I will now show you how to separate these two pieces of information using patterns in data converters. The extract step has a data converter list located in the step action view. The data converter list can be used to convert the extracted text before it is assigned to a variable. I click the plus and choose Extract to insert a data converter which can extract part of the string. A new window opens where I can configure the Extract data converter. At the top there is a pattern, and at the bottom there is a test input and a test output similar to those of the Pattern Editor. The idea with the Extract converter is to write a pattern which matches the entire input string, and then specify the subpattern to be extracted by using parentheses. By default, the entire string is matched AND extracted, resulting in identical input and output strings. If I want to exclude something from the extracted string I just have to write it outside of the subpattern. Let me precede the subpattern with '.* by ' (any text followed by "space", b, y, "space"). Now the entire string is still matched, but only the name of the author will be part of the substring, and therefore the authors name will be extracted as shown in the Test Output field. The plain text ' by ' forces the two instances of '.*' (any text) to match the date and the author name respectively. I can now close the configuration window and execute the extract step. The author name is now correctly assigned to my variable. Let me go back to the extract step and quickly demonstrate another converter which uses patterns. I remove Extract and add the Advanced Extract converter instead. Then I write the same pattern as I used before except that I make subpatterns out of both instances of '.*' (any text). The Test Output is now still 36 Kofax Kapow User's Guide the same as the Test Input. This is because Advanced Extract enables me to choose which subpattern I would like to extract by using subpattern references in the Output Expression field. In expressions, subpattern references are made using the '$' symbol followed by the reference number. Right now the expression refers to the entire matched pattern but if I change it to '$1' I only get the first subpattern, extracting the date, and if I write '$2' I only get the author name which is matched by the second subpattern. Note that it is also possible to add text, combine subpatterns, and do simple string manipulation using the expression field. For example I could write an expression which recombines the two substrings but in reverse order. For more information on expressions click the question mark next to the expressions field. Finally I would also like to recommend the Replace Pattern data converter, which replaces instances of a specified pattern in a string. Those were the final words on patterns. Feel free to review any parts of the video you found useful or go to help.kapowsoftware.com to find even more answers. Answers to Problems Problem Number Answer 1 '[0-9]' 2 '[ \n\r\t]' 3 '[^0-9]' 4 '.*' 5 'colou?r' 6 '\d{4}' 7 '.*\d.*' 8 '\D*\d\D*' 9 '(\D*|.*\d.*\d.*)' 10 '(\d)\1{3}' 11 '.*(.).*\1.*' 12 '.*?(\d).*' Snippets Click to watch a video describing how to use snippets to share steps among robots. Creating complex robots can quickly become a messy affair which hogs precious screen real-estate. Putting steps into groups is an easy way to clean up such a robot and simplify the steps it takes to reach the end result. To group steps you simply select the steps you want to group then press the group button in the tool bar above the robot editor. Furthermore you can give each group a name when you create it. This makes it easier to keep track of the specific function of the group. 37 Kofax Kapow User's Guide Collecting independent parts of a complex robot into simple groups makes the robot much easier to understand for yourself and others. Naming groups to remember their insides is essential. We could for example have a group that logs into a site, performs a complex conversion, stores values in a database, or looks something up online. Thinking about it, I actually have a login group in one of my robots which I would also like to use in some of my other robots. Now I could of course start copying and pasting this group into some of my other robots. That would solve the problem for now but if I needed to change the login procedure in the future then I would have to copy the modified group step into all my robots again. Let me introduce the snippet concept. A snippet is created and edited as a group step but is stored in a separate file and can be used as a custom step in as many robots as you want. Having the step information stored in a separate file from the robots, means that changing the snippet in one robot will change it in all of the other robots as well. This can save a lot of time and greatly simplify your robots. …In addition to containing steps, snippets can also include a list of variables used by the snippet. It could for example be a login variable containing username and password. This variable is then automatically available in every robot using this snippet. CREATING A SNIPPET Let me show you how this all works in Design Studio. Here is a robot which logs into Zoho CRM, looks up some contact information, extracts it and returns it. The login procedure of this robot has been grouped and would be perfect to have handy for other robots which are also to perform tasks on Zoho. Creating a snippet from the steps that perform the login sequence couldn't be easier. I simply select the group I already made. Then I select Convert Group to Snippet from the tool bar above the robot editor. Design Studio now prompts for a name and I choose the name LoginZoho which was already the name of the group. The snippet is then created which is visualized by the small snippet icon in the lower left corner of the box. Furthermore the newly created snippet file has been selected in the projects view and the snippet has also been opened in the snippet editor in a new tab. The snippet editor will open every time we modify our snippet in a robot. I switch to the LoginZoho snippet editor. The snippet editor looks very much like the robot editor. It has a snippet view showing the steps of the snippet, a step view showing the action performed by the active step in the snippet view and a variables view as we know it from the robot editor. Instead of the browser view, the snippet editor has a configuration view where we can write a description of the snippet. Note that in the snippet editor we can only make modifications to the configuration view and the variables view. If we want to make modifications to the steps in the snippet we will need to do that in the context of a robot editor. As you may have noticed when looking at the snippet view, the two steps which use the Login variable have been marked with error indicators, pointing out that the variable is not yet present in the snippet. 38 Kofax Kapow User's Guide Accordingly, the last thing we have to do to complete our snippet is to add the Login variable to the snippet. This will ensure that the login credentials will be available to any robot which uses this snippet. I add it by simply right clicking the variables view and choosing the appropriate type, exactly as I would have done in a robot. In the variable configuration window that now appears I have to ensure that I configure the variable to match the one in the robot precisely. The variable name is correct so I just need to check Use as Input and add default values to the attributes… then I click OK. Now we have a fully functional LoginZoho snippet. USING THE SNIPPET Let me show you how to add the snippet to another robot. Here's a new robot which I also want to perform some task on Zoho. A snippet step is inserted by choosing Insert Snippet Step Before from the tool bar. An undefined snippet step now appears in the robot. I then choose the LoginZoho snippet from the dropdown in the step view of the snippet step. The snippet has now been inserted into the robot and can be executed by clicking the end step… As seen in the variables view, the login variable from the snippet is now available for the robot to use. The small snippet icon to the left of the variable indicates that this variable belongs to the snippet. That means that we cannot edit the variable directly in the robot, but only in the snippet editor. It also means that removing the Login snippet from the robot will remove the variable as well. If we want the variable to be in the robot permanently then we have to add it manually to the robot, giving it the same name, type and configuration as the login variable from the snippet. This new variable will then take the place of the snippet variable. Now, if I make a change to the snippet in any of its instances, all the other instances will also be changed. If I for example switch the places of the Enter Username and Enter Password steps and then go back to the first robot, we see that the order of the two steps has been changed here as well. TIPS To round off this video let me give you some tips when working with snippets. TIP 1 Often you can end up with robots having huge variables containing everything needed for that robot. When creating snippets you should ultimately split these variables such that only the attributes needed in the snippet will be included in the snippet variable. In general you should try to create your types based on function rather than making them robot specific. TIP 2 To make the snippet as independent from the rest of the robot as possible, make sure to put any nondefault robot configuration directly into the steps in the snippet where necessary. In other words if you have clicked here, then here and made any changes that are important to the steps in the snippet, then make sure to click here in step view for the steps of the snippet and make the same changes. TIP 3 39 Kofax Kapow User's Guide Always document the context of the snippet by writing a description in the snippet configuration view. Date Extraction - Simple Case Click to watch a video showing how to modify the News Magazine robot to extract article dates. This tutorial will show you how to easily extract dates from any website using the Date Extraction step in your robot. The tutorial consists of two parts. This first part is a follow-along tutorial with a simple example, the second part is a real case scenario showing a trickier example. Please feel free to follow along on your computer for this first part. If you have completed the beginner tutorials you might remember the News Magazine robot. It extracts the most recent stories from News Magazine which is a site made for tutorial purposes. We want to modify the robot to also extract the date and time from the most recent articles. Start Design Studio and open the Type called post from the Beginner Tutorials folder in the default project. Add a new attribute called date and give it the type Date. Now save the Type, close the Type Editor and open the NewsMagazine robot from the same folder. Click the Return Value step in the robot view. The robot will now execute to this step. I am going to close the projects view and the source view to get some more space to work with. Scroll down in the browser view and locate the date and time given above each picture. We are going to add a step which extracts the date and time from each of the three articles. To do this we right click the tag containing time and date and select Extract > Extract Date > choose the variable post.date. A new window opens which will help us extract the date into the standard date format which is the only format accepted by the simple type Date and thus by our post.date variable. Let's look at the important parts of Extract Date Configuration window. The field Test Input shows the raw text as extracted. In the field called Pattern we will write the pattern of the date exactly as it is stated in the extracted text. Right now the pattern is "dd MM yyyy" which means that the date consists of date, month and year, separated by spaces. We see that this corresponds to the format of the date in the extracted text. Design Studio has deduced the pattern for us so if we just want to extract the date we don't have to modify anything. The field Test Output shows us the date in the standard date format as extracted from the Test Input. Let's say that we also want to extract the exact time that each article was published. This means that we have to expand our pattern to also capture this information. Keeping an eye on the Test Input, add " * hh:mm" to the Pattern. Notice that the Test Output changes to incorporate the time of day. Let me explain the pattern we added. Spaces and the colon correspond to their respective characters in the Test Input. Asterisk corresponds to "at" in the Test Input but can in general be used to represent any number of non-whitespace characters. "hh" means hours and "mm" means minutes. To get a full list of things to put in the Pattern field you can click the question mark at the top right of the window. Click OK. We have now successfully extracted the time and date from each article. Test the robot in Debug Mode to confirm this. 40 Kofax Kapow User's Guide Date Extraction - Tricky Case Click to watch a video demo showing how to extract dates when date information is spread into multiple places. This video will show you how to extract the date and time on a site that has date information spread into multiple places. It is recommended to complete the tutorial on Simple Date Extraction before watching this video. I have made this robot to extract my Skype call history. It logs into Skype and loops through a table which contains my history. The only problem is that the year is not stated directly in the date and time column but only in the blue bar above. Somehow I have to combine the two pieces of information. Luckily this is easily done using converters. Before entering the loop I add a step which extracts the text in the top bar into the variable called Year. Inside the loop I insert a new step which I choose to have the Extract action. Then I select the date tag from the first row of the table and use the yellow square button to the right of the Address Bar to use it in the Extract step. Now, to combine this extraction with the Year variable, I add a converter under the Action tab of the extract step. I choose the converter called Evaluate Expression which allows me to append the value from the variable Year to the extracted date. An Evaluate Expression Configuration window opens. The Test Input field shows the date as extracted. In the Expression field I simply write "INPUT" with capital letters, which is the extracted date. I follow this by a plus and a space in quotation marks, this adds a space after the date. Then I add another plus followed by "Year", representing the variable containing the year which we extracted earlier. Looking at the Test Output I verify that the two extractions have been combined into one text. To learn more about expressions I can click the question mark next to the Expression field. Click OK. I now add another converter, the same as used in the Simple Date Extraction Tutorial, called Extract Date. The Output of the previous converter is used as input for this converter. The configuration window opens for the Extract Date converter and I insert a new Format pattern and delete the default one given by Design Studio. Now, just as in the Simple Date Extraction Tutorial, I add the pattern to extract the date from the test input and let the converter convert the date to the standard date format. "MM dd hh:mm MM yyyy". The month is given two times but that is not a problem. The first occurrence will simply be ignored. I click OK, choose to extract into the Date variable and check that the date and time has been extracted successfully. That concludes this tutorial and demo on extraction of dates. API Click to watch a video, which takes you through the creation of a robot that uses JSON and a REST call to access the LinkedIn API. 41 Kofax Kapow User's Guide Introduction This video will take a look at how to utilize the power of REST web services, APIs, OAuth and JSON in the creation of a robot. Specifically we will go through the process of automating the posting of a share to a LinkedIn account based on content extracted from another website. This process involves the following: 1. creating an app through a LinkedIn developer account 2. looking at the LinkedIn documentation reference and understanding how the share API works 3. creating a robot which extracts from News Magazine and posts to LinkedIn through a REST call to the LinkedIn share API, and finally 4. scheduling execution of the robot and adding users to it from the Management Console. After watching, you should be able to access and call the LinkedIn API. Even if you are only interested in other APIs, this video will still serve as a fine introduction. OAuth and the LinkedIn API Before we can start working on the robot in Design Studio, we need to get access to the LinkedIn API. This is only possible with a developer account, but fortunately it is quite easy to get one. Start by going to developer.linkedin.com and sign in in the upper right corner. In the popup window that opens, type your username and password and allow the LinkedIn Developer Network to access your account. Once you have successfully logged in, click on the link to API Keys, in the dropdown in the upper right corner. This will give you a list of the applications associated with your account. The list should be empty unless you have already created some LinkedIn applications. Add a new application, and fill in the form. Apart from filling in all the required fields, you should also check off any permission that your robot might need. Since we need to post to the users account, we will check the rw_nus permission. When you are done, click Add Application at the bottom. LinkedIn now gives us back a page showing the OAuth credentials needed to access their API. We leave the page open so we can have access to these credentials when we need them. In a new tab, we open developer.linkedin.com again. This time we go to the Documentation dropdown box and click REST APIs. We need to find the part of the API which lets us share something to an account on the user's behalf. We choose Share and Social Streaming in the Documentation Menu on the right, then choose Share API from the submenu. Now we are on the page for documentation of the Share API. Scrolling down a bit, we see what a sample payload for the Share API looks like. The example is in XML, but we will be posting using a JSON payload instead. The translation from XML to JSON is pretty straight forward as we will see. Above the example is a table which shows all the fields available to the Share API. It is possible to set a comment, a title, a url, an image and a description. JSON and REST call Now that we have OAuth credentials for the LinkedIn API, it is time to start building our robot. We create a new robot called LinkedInShare. Then we add a variable of the type OAuthCredentials, name it credentials and mark it as an input variable. It is important to make it an input variable so we will later be able to schedule the robot in the Management Console. Then we give it the following default values which will let us call the LinkedIn API: We write LinkedIn in the serviceProvider attribute, copy the API Key to the consumerKey attribute, Secret Key to consumerSecret, OAuth User Token to accessToken, and OAuth User Secret to accessTokenSecret. As you might notice, there is no strict naming convention these four tokens. Hence the names differ between LinkedIn and Design Studio. Then we click OK. We now need to create a new type: One into which we can extract the front page article from a website called News Magazine. We create a new type and call it Share.type. For this type we should name each attribute according to the input fields specified by the Share API. Going back to the documentation of the 42 Kofax Kapow User's Guide Share API, we see that the fields we are worried about are called title, submitted-url, submitted-image-url, and description. In Design Studio we add the four corresponding attributes to our type: title as a short text, … submitted_url as a short text, … submitted_image_url as a short text, … and finally description as a long text. Note that we use an underscore instead of a hyphen since hyphens can't be used in valid attribute names. Luckily, LinkedIn does not distinguish between the two. That's it! We save the Share type. Going back to the robot, it is time to start the fun part, which is building the robot. We start by adding two variables, one of the type we just created, and one of the simple type JSON. The JSON type variable should be able to contain the JSON payload which will be used in the API call. Starting with the jsonPayload variable, we add it to the robot and edit the default value of the variable to reflect a payload template. I have made the template ahead of time. You can copy the template for the payload from the text associated with this video: { "comment" : "News Magazine article of the day", "visibility" : { "code" : "anyone" } } The payload template contains a default value for the comment field. This could be created dynamically if you wanted to. It also contains a value for the visibility of the share with the code "anyone". This means that anyone, not just connections, will be able to see what we post to the user's profile. Save the variable. We then add a variable of the type Share. Now all we need to do, is to load News Magazine, extract from the front page story, add that content to the payload and call the LinkedIn API with a Call REST Web Service step. News Magazine is most easily loaded by inserting a snippet step and choosing the snippet LoadNewsMagazine for it. This is one of the snippets that comes with the default project when you install Kofax Kapow. Executing the snippet step, we are able to see the front page of News Magazine. From the featured article on the front page, we extract the title, url, image url and description into the share variable. The url is extracted from "Continue Reading" and the image url is extracted by right clicking the image and extracting the url. … To build the payload we now have to open the jsonPayload variable. This can be done by right clicking it and choosing insert step>>open variable. The value is now showed in the JSON view. Click the plus icon in the toolbar above the JSON view to expand all lists and objects and view the entire content of the variable. Inserting the content from News Magazine into the payload is very easy. Simply right click the comment property and choose Modify>>Insert>>After. In the Name field we write "content". For the Value field, choose Generate From Variable from the value selector on the right. Now we can choose our share variable in the field called Variable to Format. Click OK to finish. The JSON payload should now have been correctly formatted for our REST call. We insert a new action step as the last step before the end step and choose for it the Call REST Web Service action. All we need to do is configure this step and the robot will then be ready to be deployed. Here is how to do it. From the Share API documentation we copy the URL and paste it into the URL field. The request method is POST, which is also stated in the API documentation. 43 Kofax Kapow User's Guide From the dropdown just below the request method, we choose that we want to specify the raw body of the request rather than individual parameters. For the request body we select variable from the value selector and choose our JSON payload variable. Correspondingly we choose application/json as the content type. Going down to the bottom of the Step Action, we need to specify credentials for the REST call. We have already prepared these. Choose OAuth as authentication type, and select the OAuthCredentials variable. That was the final step in finishing the robot and it should now be ready to for deployment. First however, we will run it in debug mode to check that it works as expected. We switch to debug mode and run the robot. If it is successful, there should not be any error messages. Next, we log into LinkedIn to check that the share has been posted successfully. It may take a while before the share appears. Once we have ensured that everything looks good, it is time to upload the robot to the Management Console in order to schedule it and add multiple users. Schedule the robot Going back to Design Mode from Debug Mode, we upload the robot to the Management Console and click the link that appears to the Management Console. Since our robot uses the OAuthCredentials type as input variable, setting up the scheduling is a bit different than you might be used to. Here is how to do it. In the repository, we go to the OAuth section. Here we add a new application, namely our LinkedIn application. We name it "LinkedInShare", choose LinkedIn as our service provider, copy our application API Key to the consumerKey field, and our Secret Key to the consumerSecret field. We leave the Scope and Callback fields as they are and click save. Then we add a user. We call the user "user1", click next and click the link to authorize the user. In a new window, our application will now request permission to access whatever LinkedIn account we are logged in to. Allow access and the Callback page will confirm that permission has been granted. Close the window or tab to go back to the Management Console. Now we click next, and then finish, to finally add the user to our application. Multiple users can be added in this way, but let's keep it simple and stick to just one user at the moment.. Clicking on the Schedules tab, we are now going to create a schedule for the robot. We configure the schedule, give it a name, set it to run daily, and add a job to the job list. We choose to add a single robot and select the LinkedInShare robot. In the last step of the dialog, we choose the user we created for OAuth and click finish and save the schedule. The task has now been completed. We have set up an automated process which uses the LinkedIn API to post daily updates to a LinkedIn profile. You should now be able to use the LinkedIn API. If you want to learn more about the using OAuth to access sites like Salesforce, Facebook, and Twitter, please consult the documentation at help.kapowsoftware.com, where you will find a section called OAuth in the Management Console User's Guide. 44 Chapter 3 Design Studio Design Studio is the application for creating robots and types. In Design Studio, you can also debug your robots and create database tables for types that need to be stored in databases. Design Studio, an integrated development environment (IDE) for robot development, is all you need to design robots and types. Robots are programmed in an easy-to-understand visual programming language with its own syntax (structure) and semantics (meaning). To support you in the construction of robots, Design Studio provides powerful programming features, including interactive visual programming, full debugging capabilities, an overview of the program state, and easy access to context-sensitive online help. Design Studio also lets you create the types that are used by robot variables for data extraction and input. With Design Studio Type Editor, you can design types that are modeled after real-world data. In the most common case, a type is designed to hold the data that a robot extracts from a data source. Organization The Design Studio section of help is structured as follows: First, you are introduced to the essential concepts of Design Studio. Then you are taken on a tour of the user interface and provided with an overview of the core building blocks of any robot. With the basics firmly in place, we get to the tutorials that show you how to use Design Studio to create robots that do something useful. The tutorials get gradually more advanced until you are ready to create robots that perform the tasks defined by you. The tutorials are the meat and bone of this section of help and it is important that you master them before proceeding. Before You Read On Before you proceed, we recommend that you read Introduction to Kofax Kapow, which will introduce you to Design Studio and the context it is used in, and take you through basic tutorials. Note To access tutorials, you must have access to the Internet. To work with Design Studio, you should have a basic understanding of programming, HTML, and JavaScript. Other Resources Additional information on Design Studio is available in the reference documentation. 45 Kofax Kapow User's Guide Introduction to Design Studio Design Studio is a programming environment for creating robots and designing types. Robots are created using a special-purpose programming language with its own syntax and semantics. Like other programming environments, Design Studio uses several concepts that you, as a robot designer, must understand to fully comprehend the workings of Design Studio. The purpose of the introduction is to define the most important concepts and we recommend that you refer back to this section whenever necessary. The Design Studio concepts becomes clearer as you explore Design Studio and start creating robots. Robots The most important concept in Design Studio is a robot. A robot is a program designed to accomplish some task involving a data source, usually a web site, but it could also be an Excel document or a database. Typically, one robot is written per task per data source. For example, you would create one robot for extracting news from http://cnn.com, another robot for extracting news from http://yahoo.com and yet another robot for extracting product information from an online product catalog. Basically, a robot can be programmed to do (automatically) everything you can do in a browser, and to extract data from a database or an Excel document to combine with data stored in a database or file. Robot Execution Mode Kapow Design Studio supports two design-time robot execution modes: Minimal Execution (Direct) and Smart Re-execution (Full). This topic provides details about the two modes. When creating a new robot, you can select execution mode in the new robot wizard. Use the Design Mode tab of the robot configuration to view or change the execution mode. Note that the choice of robot execution mode only impacts the execution in Design Mode, and not in Debug Mode or at runtime in RoboServer. Minimal Execution (Direct) The Minimal Execution (Direct) mode is the traditional Design Studio execution mode. All robots written in versions prior to 9.5 will use this execution mode, which is also the default mode for new robots. When you click a step in Minimal Execution mode in the robot graph, Design Studio takes the shortest direct path to that step, skipping any previous branches and iterations that are not on the direct path. Consider the example below: 46 Kofax Kapow User's Guide During runtime execution, the robot would normally execute steps A, B, C and D before reaching step E. But in Design Mode, clicking step E result only in the execution of steps A and D. Similarly, if the step resides inside a loop, only the selected iteration is executed. Since the iteration counter is set to 3, clicking step C cause only step A, B and C to be executed once, where step B selects the third iteration. The Minimal Execution mode is optimized towards executing as few steps as possible. This mode is useful when you have large robots and steps that take considerable time to execute, such as steps that interact with complex websites. Generally, we recommend Minimal Execution for most data collection use cases and for robots that perform significant interaction with external sites. The drawback of Minimal Execution mode is that it requires user assistance to select the path to a given step whenever it cannot execute directly to the step using the default path; for example, try steps in a path may prevent a robot from following the topmost branch. See the following example. When clicking step C, Minimal Execution mode is not able to proceed if the test fails in the Test Value step. In this case, the user must explicitly click the bottom branch of the try step first, to guide the execution path towards step C. Smart Re-Execution (Full) In Smart Re-execution mode, the way that the robot is executed in Design mode is similar to the way it is executed at runtime or in Debug mode. As an example, when you click step C in the following robot, it automatically executes through the bottom branch of the Try construct when the test fails in the Test Value step: The blue exclamation mark icon on the Test Value step indicates that the error handling to Try Next Alternative was triggered. With loop steps, all iterations up to and including the selected iteration are executed when clicking a step inside the loop. In the following example, clicking step C causes execution of three iterations of the loop. 47 Kofax Kapow User's Guide Further, if clicking a step in a branch below a loop, all iterations of the loop are executed. As an example, see the following robot. Clicking step D causes execution of step A and repeated execution of steps B and C (as many times as there are iterations in loop B) before finally stopping at step D. The Smart Re-execution mode is particularly useful when working with global variables, and when you have subsequent steps in the robot that depend on accurate variables. This mode may be useful for building a payload for a web service (REST or SOAP) call, or constructing an Excel document. The XML or Excel document that is being populated resides in a global variable, while its content is added during the execution of a loop. In a branch below the loop that populates the document, the robot takes the entire document and posts it to a web service or similar. In this case, the Smart Re-execution mode makes it easier to build the robot, as it ensures that the document is populated when testing the web service call in Design Mode. In Smart Re-execution, the interaction with the external world in the form of websites, databases, or web services is cached. Caching avoids re-execution of steps unless the prerequisite for storing the execution result has changed (such as a variable that determines which URL to load). Smart Re-execution has a higher memory footprint than the Minimal Execution mode. The Smart Re-execution mode is the only mode that supports Device Automation workflow. We do not recommend Smart Re-execution mode for large robots with significant interaction with the external world, or for long-running robots. The execution time as well as memory usage are too high in these cases. 48 Kofax Kapow User's Guide To cut down on the execution time while designing the robot, you can right-click a branch and disable it in Design Studio. A similar setting can be applied in Debug Mode. Additionally, you can disable the branch based on a specified condition in select iterations. Connection configuration The Design Mode tab of the robot configuration also features an option "Avoid External Re-execution". When checked, it is ensured that steps are never re-executed, even when the cached result of the previous execution cannot be used. In this case, you can still edit the robot, but without a current input state to work on. Use this option only to meet requirements for interaction with the external word to avoid re-execution (for example, if re-execution would result in incorrect or duplicate data in a partner's system). Important Some step actions are not available in the Smart Re-execution mode. For a list of unavailable steps, see the "Execution Mode" section in the Kapow Limitations topic. The Robot State When a robot is executed, it works on a robot state, which consists mainly of four elements: • Windows • Variables 49 Kofax Kapow User's Guide • Cookies • Authentications The Windows element corresponds to the currently open windows, each containing a page. This page could be an HTML page, a spreadsheet, an XML page etc. The page has a given Page Type depending on what type of page is loaded into the window and the look of the Page View and the steps you can insert in your robot depend on this type. At least one window is always open, and one window is marked as the current window. The variables element contains the current values of the variables. The cookies and authentications elements are the HTTP cookies and authentications, respectively, received during communication with a web server. Steps A robot is made up of steps, which are building blocks in a robot program. There are four types of steps: • Action • Try • Group • End A step works on a robot state and processes it according to the configuration of the step. A step has an input robot state and generates an output robot state. The only exception is the End step. End steps mark the end of a branch in a robot, but not the end of a robot. For example, the robot does not necessarily stop execution after an end step. End steps are the only steps in a robot that do not have outgoing connections. Steps may have properties, such as a step name, a list of tag finders, a step action and error handling. While Action steps have all these properties, other types of steps only have some. The step name provides a symbolic name for the step, such as "Extract Headline" and "Load Search Page." In the preceding robot, the step name is "MyStep". The finders find the elements (HTML/XML tags or Excel cells) in the page that the step action should work on. Some step actions require a single element, whereas others can handle more than one element. Some step actions accept no elements at all. There are two kinds of finders: Tag Finders that find tags in HTML or XML pages and Range Finders that find cells in Excel pages. The step action is the action that the step performs. The action is the "heart and brain" of the step, and it is the selection of the right step action that is the challenge of robot writing. For example, an Extract action can extract the text from a tag in an HTML page and store it in a variable. A Click action can load the URL residing in an -tag and replace the page of the current window in the robot state with the newly loaded HTML page. An action usually changes the robot state. For example, the Extract action changes the variables, and the Click action may change the pages/windows, the cookies and the authentications. 50 Kofax Kapow User's Guide A step can be executed. A step that is executed accepts a robot state as input and, by applying the finders and step action in turn, produces an output robot state. The output robot state is then passed to the following step and becomes its input robot state. Some step actions are "termed loop" actions and steps having such actions are called "loop steps." A loop step may generate zero or more output robot states, and cause the following steps to be executed once for each of them. You can group steps together in expandable Group Steps. The figure below shows an example of an expanded Group step with a collapsed Group step inside it. A step is valid if it is properly configured so that execution can be attempted. For example, if a step has no action, it is invalid since execution cannot be attempted. A step definition also specifies error handling. Connections and Execution Flow Use connections to determine the execution flow between steps. Note Examples in this topic are based on the Minimal Execution (Direct) design-time execution mode. Consider the following simple robot: This robot consists of three steps: Step A, Step B, and Step C. Assuming that no errors occur, and that each step generates exactly one output robot state, the robot is executed as follows: An initial robot state is generated and used as input to Step A (being the first step). Step A produces an output robot state. This output robot state is the input robot state of Step B. Similarly, Step B produces a robot state, which is the input robot state of Step C. Once Step C has executed and produced an output robot state, execution completes. In short, the execution of steps is described as follows: "A, B, C." Sometimes, a step generates no output robot state when executed. This happens when an error or a test step causes execution to continue somewhere else in the robot (see Conditions and Error Handling). Steps containing a loop action may process the input state several times, each time outputting a distinct robot state. Consider the following robot where step B contains a loop action: 51 Kofax Kapow User's Guide Assuming that there are no errors or test steps, that step B outputs three robot states, and that all other steps output exactly one robot state, the steps are executed in the following order: "A, B[1], C, D, B[2], C, D, B[3], C, D", where B[ N ] refers to the N th iteration of the loop action contained in step B. Note that the output robot states by step B are different robot states: each iteration will output a new robot state. Hence, step C will receive a new input robot state each time it is executed. See the Branches, Robot States, and Execution Flow tutorial for more information. A step can connect to more than one step. This is called "branching". Consider the following robot: In this robot, step A is followed by a branch point, where the connection splits out in two branches. One branch consists of step B and step C, and another consists of step D and step E. All branches coming out of a branch point are executed, one after another. Therefore, assuming that no errors or test steps change the control flow and that each step generates exactly one output robot state, the preceding robot is executed as follows: A, B, C, D, E. However, it is important to note that step B and step D each receives a copy of the same output robot state produced by step A. Branches can merge, and in complicated ways. Consider the following robot: This robot illustrates how connections can be explicitly ordered. In this robot, the branches of step D are executed in the order specified by the numbers: step E is executed before step C. If an order is not specified (by numbers), connections are executed top-down. Thus, assuming that there are no test steps, that no errors occur, and that each step generates exactly one output robot state, the robot is executed as follows: A, B, C, D, E, C. The first time step C is executed, it receives the output robot state produced by step B; the second time step C is executed, it receives the output robot state produced by step D. Sometimes you want to select (execute) only one of several branches, depending on circumstances. The Conditions and Error Handling topic shows how to do this. 52 Kofax Kapow User's Guide Conditions and Error Handling A robot may use different approaches in different cases. The cases may be distinguished either based on explicit tests; by evaluation of conditions, or because errors occur and need to be handled. Note Examples in this topic are based on the Minimal Execution (Direct) design-time execution mode. Conditions change the flow of execution based on the content of the input robot state (such as the presence of a particular tag in an HTML page). Error handling is about changing the flow of execution when particular errors occur (for example, some anchor tag is not found on the HTML page as expected and cannot be clicked). Often a situation can be seen both ways: An anchor tag should be clicked if found (this is a condition), or the robot can try and click it to handle the error (if it is not found). In some cases, what is commonly thought of as a condition is too complex to be written up as such (for example, a condition saying "if this particular page can be loaded without error"). In such a case, try and load the page and treat any error as an indication that the condition failed. Other errors are signs of genuine problems with the robot or the web site being accessed. For example, the web site may be down and cause a page loading error, or a tag finder might fail to find a needed tag due to a dramatic page layout change of an HTML page. A particular error may be considered a failed condition in some circumstances, and a genuine error in other circumstances. The interpretation depends on the robot. Because of this blurred boundary between conditional execution and error handling, Design Studio provides both features in a unified way. For every step, you can configure what to do in case an error occurs. Furthermore, steps with a test action (based on a condition of some sort) reuse the same approach, meaning that if the condition is not met, the (default) action is applied as if an error occurred. For each step in the robot, you can configure the desired reaction to errors. Two useful error handling options are described here; see How to Handle Errors for information on the other options. The first option is closely linked to the Try step. The Try step is similar to a branch point because it may have several branches going out from it. It differs from a branch point because branches beyond the first one are executed only if a step on the preceding branch encounters an error which it handles based on the Try Next Alternative option. Consider the following robot and assume that each ordinary step is expected to output exactly one robot state: The icon indicates that step B is configured to handle errors by "Trying Next Alternative." If Step B executes successfully, step execution is as follows: "A, T, B, C." Because the first branch going out from T executes without error, the second branch is not executed at all. 53 Kofax Kapow User's Guide If, on the other hand, Step B encounters an error, then the execution of steps is as follows: "A, T, B, T, D, E." After the error in Step B is handled, execution does not continue at the following step, but instead at the beginning of the next branch going out from the Try step. Each branch from a Try step represents one possible way to proceed from that point. Steps near the beginning of each branch probe if execution along the branch is a viable approach (and otherwise effect a "Try Next Alternative"), while later steps do the actual work when the branch turns out to be the right one for the case at hand. The probing steps near the beginning of a branch may be either test steps, or any kind of steps that, if they encounter an error, indicate that this branch is not the way to proceed. There may be any number of such branches going out from a Try step. As with ordinary programming languages such as Java, JavaScript, C# or similar, the preceding robot is similar to an "if-then-else" construct: The first branch after the Try step contains the condition (the "if" part) and the "then" part, while the last branch contains the "else" part. Should there be more than two branches, then the ones between the first and the last ones are like "else-if" parts. If the first branch attempts to do some action that may error, the example can also be likened to a "trycatch" construct: The first branch is the "try" part, while the second branch is like the "catch" part. Another error handling option, Skip Following Steps, provides a more compact way of expressing a common special case, which is exemplified by the following robot. The step that can encounter an error is the first one on the first branch, and the second branch does nothing. The effect is to skip execution of the steps after step B if it encounters an error. The same effect can be achieved without the Try step by using the error handling option "Skip Following Steps" (which is the default), in the following way. Location and Location Code When an error is handled, it is possible to report it back to the caller of the robot, or to log it. In both cases, a message is included that briefly describes the error, together with a location and location code for the step that encountered the error. The location of the step that encountered the error is the list of steps (including iteration numbers) necessary to execute to reach that step from the first step. Consider the following robot. 54 Kofax Kapow User's Guide If Step C reports an error on the second iteration of Step B, the location is written as: "step A - step B[2] - step C." Note that the location contains the step names and iteration numbers, separated by hyphens. Branch points are omitted. The location code is similar to the location, but the name of each step is replaced by a unique identifier for that step, thereby avoiding name clashes. For the preceding location example, the location code may be: {a-i1-a}. Use the location code in Design Studio to go directly to the step that reported the error (using Go To Location on the Edit menu). Important The iteration number in the location and location code is 0 indexed, so the first iteration is: {a-i0-a} Snippets A snippet is a group of steps that can be reused in several robots. A snippet is maintained in a file separate from the robot. Whenever the contents of a snippet is changed in one robot, it is automatically updated in other robots that uses the same snippet. A snippet is inserted into a robot using the Snippet step, and edited in-line. Snippets contents cannot be edited without being inserted into a robot. For additional information, see the Snippets tutorial. The Snippet step inside a robot is in many ways similar to a Group step. Although, the steps inside a Group step are part of the robot, the steps inside a Snippet step are maintained in a separate file and can be reused in other robots inside the same project. A robot is incomplete and cannot execute if a snippet that it references is not present in the project. After selecting a group of steps to convert to a reusable snippet, click "Create snippet from selection." If only a single group step is selected, it can be converted to a reusable snippet by clicking "Convert snippet to group." A snippet can be easily embedded into a robot by clicking the "Convert snippet to group" icon after selecting a snippet step. A snippet can also define a set of variables included in the set of variables of any robot that uses the snippet. A snippet can have a description. This is edited in the Snippet Editor and is shown on every occurrence of that snippet in robots. Variables and Types Variables and Types are important concepts in Design Studio. Every variable can be associated with a default initial value that it retains unless the robot explicitly reassigns it, which it often will as values are extracted and manipulated during the execution. Most robots output the values of variables, by returning them to the caller or inserting them in a database. Robots 55 Kofax Kapow User's Guide can also take input values which are assigned to specific variables marked as receiving their values from input. These are called input variables. You define each variable as either complex or simple. Simple Variable A simple variable does not define any attributes, but only represents the type of a single value. Thus, a variable of a simple type contains a single value, for example a text string, and is referred to only by its variable name, such as Username. Simple types are built-in and cannot be edited, or created. • Useful when extracting temporary data or as global counters. • Commonly used as temporary variables, internal to the robot. • You cannot use a simple type for input variables. • You cannot output the value of a simple type. Complex Variable A complex variable defines a set of attributes. Each complex variable denotes several (named) values. We generally refer to each attribute such as "title" as a separate variable such as "Book" and denote its value using the fully qualified attribute name, such as Book.title. You can create complex types within Design Studio to suit your needs. Complex variable values are output in various ways. For example, a robot extracting news from a web site might output the values of news variables; each news variable would have a complex type with attributes such as headline, bodyText, date, and author; and each news value to output would comprise a possibly unique subvalue for each named attribute. For robots containing input variables, they must be specified as part of the robot's input with values assigned to the input variables. For example, a shopping robot that orders books at http://amazon.com might depend on input values containing user and book information. These might be assigned to two input variables in the robot called "user" and "bookInfo" of type "User" and "BookInfo." The following figure shows how a robot accepts input values and generates output values. The figure shows robot input-output. Input values are assigned to input variables, and the values of some variables are output. Only variables of complex types can be assigned from input or have their values output. Libraries and Robot Projects Robots and types are organized in libraries. A library is a collection of robot definitions, type definitions and other files needed to execute the contained robots. A library serves as the deployment unit for robots. 56 Kofax Kapow User's Guide Use a library to bundle robots and their required files when you want to distribute and deploy the robots in a runtime environment, such as RoboServer. In Design Studio, you can work on one or more robot projects at any time. The purpose of a robot project is to develop a robot library. A robot project contains the robot library that you are developing a given set of robots in, as well as other files that are useful for your work on the robot library. Files placed in the library may also be accessed by robots using a special library protocol. Thus, a robot project is what you work on when you are developing robots, and a robot library is how you distribute and deploy your work. Shared projects are deployed on a Management Console and connected to a project on your local Design Studio computer. Management Console projects can be shared between several Design Studios. The Shared Projects View provides visual indication of the status of the shared project files as well as tips with descriptions. Naming policy Kapow imposes the following naming policy, which applies to project names, schedule names, folder names (paths), and folder items including robots, types, snippets, and resources uploaded through the API. • • • • • Illegal system characters are allowed, but they generate a warning message. System reserved words are allowed, but they generate a warning message. Empty names are not allowed, and they generate an error message. Names that exceed 243 characters are not allowed, and they generate an error message. Names with special HTML formatting are allowed, but they generate a warning. The HTML formatting is removed. Note the use of a period "." in file names: • Files and folders starting with a period “.” are treated as hidden (hence all files in hidden folders are also hidden) and they are not shown on the project tree. • Hidden files are not included in project synchronization. • Robots, snippets, types, texts, and database mapping files cannot have names starting with a period ".". • Folders cannot have names starting with period ".". Name Conversion Algorithm Typed name Result 'aaa ' 'aaa' Note ' ' denotes the start and end of the string. Note The HTML tags and are removed. The space before and after the actual folder/project/ schedule name is also removed. 57 Kofax Kapow User's Guide Typed name Result '< t> hello ' '< t> hello' Note The tag < t> is kept because it is not a valid HTML tag. The space after the actual name is removed. 'com1 /*** /& @' 'com1/***/& @' Note Because "com1" is a reserved word for some operating systems, it generates a warning; however, it is a valid name for a folder, project, or schedule. An asterisk "*" is an illegal character. Kapow accepts it as a valid name but generates a warning. Note that entering a name with reserved words and illegal characters can result in errors when other users download the project to local computers where the operating system does not support the names as valid file names. HTML entity numbers such as & are converted to actual characters. Design Studio User Interface This topic introduces the Design Studio user interface and begins the tour to the following elements (or others): • Menu bar • Tool bar • My Projects view • Shared Projects view • Databases view • Editors view • Robot Editor • Type Editor • Text Editor We recommend that you start the Design Studio to follow the user interface tour. Note, however, that the tour explores the Design Studio user interface as it appears at startup if you do not create or load a robot. To view the Design Studio main window, you must have a valid, activated license. See the Kofax Kapow Installation Guide for details on licensing. 58 Kofax Kapow User's Guide Menu Bar The menu bar is located at the top of the Design Studio window. 59 Kofax Kapow User's Guide The available menus and included items are based on the type of file that is open in the Editor view. The following menus are always available even if no file is open (some items may be disabled): • The File menu includes items for manipulating files, projects, and so on. • The Options menu includes items for changing default settings and defining proxy servers or database connections. • The Window menu includes items to change the layout of the user interface, such as Reset Layout. • The Help menu includes links to the online reference help, documentation, and technical support information. As soon as you open a file, such as a robot, the Edit menu is added to the available menus: • The Edit menu offers a range of edit actions that you can perform on the opened file. The available actions depend on the type of the file, but it always contains the Undo and Redo actions. If you open a type or a robot file, one more menu becomes available: • The Tools menu lets you perform tasks related to the type of the file, such as generation of database table (for types), or deployment of robots to the Management Console (for robots). If you open a robot file, three more menus may be available: • The View menu lets you perform actions on the view or open additional views that are not open by default. • The Debug menu contains actions related to the debugger. • The Breakpoints menu contains actions related to the breakpoints in the debugger, such as adding and removing breakpoints. This menu is only available when the Robot Editor is in debug mode. 60 Kofax Kapow User's Guide Toolbar The Toolbar buttons let you perform many actions that are also available on the menus. The available buttons change, depending on which editor is active in the Editors view. Icon Description Open Project Save All Files Configure Robot Synchronize All Stop - Escape Refresh Undo Redo Cut Copy Paste Before Delete Insert step before selected step 61 Kofax Kapow User's Guide Icon Description Insert step after selected step Add branch from selected step Group Ungroup Create snippet from selection Convert snippet to group Move step or connection up Move step or connection down Expand All Collapse All Switch to Debug Mode Start Debug from current location Download robot from Management Console Upload robot to Management Console My Projects View The My Projects view is located under the toolbar icons in the Design Studio main window. 62 Kofax Kapow User's Guide The My Projects view shows an expanding/collapsing tree structure representing the robot projects that are open in Design Studio. Click a + or - in this tree to expand or collapse the corresponding subtree. This view can contain as many opened robot projects as you like. In the My Projects view, you can right-click to open a context menu to perform various actions, such as creating a new robot in a folder, or opening a previously saved robot. Shared Projects View The Shared Projects view is located in the lefthand pane of the Design Studio main window under the My Projects view. You can rearrange the views by dragging them to different locations. The Shared Projects view shows an expanding/collapsing tree structure representing the robot projects for the Management Consoles that you are connected to. If any projects in the Shared Projects view are shared with the Design Studio on your computer, both project views will contain those projects. Depending on the status of the files in the shared project, downloaded, updated, and deleted files have a different appearance. You can synchronize your local project with the Management Console project using different strategies. The following table shows project files with different statuses. The Shared Projects view also provides tips and explain synchronization problems. Icon Description Meaning Dimmed robot icon and name An object in a shared project exists on the connected Management Console, but has not been downloaded to your Design Studio. Normal icon and object name An object in a shared project is in sync with remote Management Console. Normal icon with object name crossed out The object is deleted in the project on your computer. Normal icon with name in bold The file has been changed locally and needs to be synchronized. Icon with a plus sign and name in bold A new file in your local project is not yet uploaded to the Management Console. Object icon has a yellow sign with an exclamation mark Conflict exists between your local copy and remote project. For example, the object was deleted on a remote Management Console. When synchronizing, select how to resolve the conflict. Databases View The Databases view shows databases for Design Studio and any connected Management Console. 63 Kofax Kapow User's Guide To configure connections to the Management Console, navigate to Settings > Design Studio Settings > Management consoles The databases are fetched via database mappings in the Management Console. To have the databases displayed in the Design Studio Databases view, database mappings must exist for the cluster databases you want to share with Design Studio users. Unmapped cluster databases are not displayed in Design Studio. Important The database mappings, types, and drivers are fetched from a Management Console only when the connection between the Management Console and you copy of Design Studio is established or refreshed during the following events: • Adding a Management Console connection to Design Studio • Starting Design Studio with an existing Management Console connection • Refreshing the Management Console connection (to refresh, select the Management Console node in the Databases view tree and click Refresh). Editors View Use the Editors view to edit your robots and types. You can have many editors open at the same time, but only one editor is shown. The editors are shown as tabs at the top of the Editors view and you can click a tab to switch to another editor. There are three kinds of editors: • The Robot Editor in which you edit a robot. • The Type Editor in which you edit a complex type containing one or more attributes. • The Text Editor in which you edit a plain text file. 64 Kofax Kapow User's Guide Robot Editor Use the Robot Editor to edit robots. When you open a robot, it appears in a new Robot Editor placed in a new tab on the Editors view. The Robot Editor has two modes: design (default mode) and debug. Select a mode by clicking a mode button in the left corner of the Robot Editor. Depending on which mode you select, the appearance and availability of options may vary. Each view consists of several subviews. For the design mode these are the following: • Robot View • Windows View • Step View • Variables View • Frames View Robot View The Robot View is located at the top of the Robot Editor under the tabs. The Robot view shows you the robot program: the steps and connections that make up the robot. In this view you navigate the robot steps. Select a step to edit its structure such as delete, move, or connect steps. Current Step In the Robot View there is a notion called a "current step". The basic idea is that the partial robot that you are building is actually executed while you are building it. The current step marks the position in this execution and the Studio shows the state in the Page view and the Variables view. The current step is marked in green. Click a step to execute the robot up to that step. The selected step becomes the current step. While the robot is executing, the step you clicked is shown in yellow. When execution reaches the step, it becomes the new current step and appears in green. If execution cannot reach the clicked step (for example, an HTML page does not load), the execution stops at the valid step, which becomes the new current step. If you click a step the robot has already executed, no execution 65 Kofax Kapow User's Guide occurs, but the new step becomes the new current step. You always configure the current step in the Step view. Current Execution Path The Current Execution Path is the path in the robot that the execution performed to get to the current step. The robot continues on this path until it reaches a branch or an end node. The current execution path is marked by a darker color on the connections. You can change the current execution path by clicking a connection, which will result in the connection being included in the path. Select Items To select a series of steps or connections, press and hold the Ctrl key and click the items. You can also hold down the left mouse button and drag it over the steps to select. Click anywhere outside the robot to deselect currently selected steps and connections. When steps or connections are selected, you can apply actions to them. For example, to insert a new step, select a step and click the Insert Step After on the toolbar. You can also right-click a step or connection to select an action from a list. Note When you right-click a step or connection, it is automatically selected. Edit Actions The Robot Editor lets you perform a long range of actions on steps and connections. These include standard editor actions such as copy, paste, cut and delete, and actions that affect the execution of the robot in the design view, such as changing the iteration of a loop. You can perform actions on either the current step (if no other step is selected), selected steps or selected connections. Perform an action by clicking the corresponding toolbar button or by using the context menus on the selected elements. You can configure another step by selecting it (by Ctrl-clicking it or dragging a selection box around it) and pressing the F2 key or selecting Configure Step on the context menu. For more information, see General Editing. Windows View The Windows view is located under the Robot view in the Robot Editor. In the Windows view, you can see a part of the current robot state: the part of the robot state that has to do with loaded pages. The state shown is the input state to the current step. In the Windows view, you see the Page views of the windows in the current robot state. When loading from a URL, several windows may be opened, each containing a page. The current window is marked with an arrow. If the opened page contains non-HTML content, you can preview the page depending on the type of the content. Use the Preview button to change the type of the content. You can preview CSV, JSON, text, Excel, XML, and binary content and apply step actions to them. If you use the Classic browser engine, for each window, the Page view is split into several sub views depending on the type of the page. For example, if the loaded page is an HTML page, the Page view has sub views. There are five types of pages: HTML, XML, JSON, Excel and Binary. HTML and Binary use the same view and the other page types use their own specialized Page views. Note To view XML content with the applied XSLT transformation in the windows view, select Configure Robot > Default Options: Configure > Legacy tab > Format Handling: Classic loading and clear the Convert XML to HTML option. 66 Kofax Kapow User's Guide To see the Cookies view of the state of the current step, you can open the Cookies window from the View menu. Cookies are added to this list as the robot loads web pages that use cookies. Similarly, you can open the Authentications window from the View menu to see the authentications of the current state. Step View The Step View shows the configuration of the current step. Click the tabs to view and edit the following properties: • Basic: Includes the name of the step and any associated comments. Steps with an attached comment are shown with a name in bold in the Robot View. You can rest the mouse pointer on a step to view the comment. • Finders: View and configure the list of finders of the step. You normally configure the finders by rightclicking an element in the Page view. See Using the Tag Finders. • Action: View and configure the action for the step. For a description of the available actions, see Step Actions and Data Converters. • Error Handling: See how the current step handles errors. See Handling Errors. Variables View The Variables View includes a list of variables. When you select a variable from the list, the associated details appear on the righthand side of the view. The view shows the variable values for the current step of robot execution, and cannot be edited. • Right-click the variables list to access a list of variable types. You can add or remove variable types using this list. You can also remove the selected variable using this list. • Click Edit to modify the initial variable values, or double-click an item in the variable list. A variable view similar to the Variables View window appears. This window displays the values of the variables before any step has been executed, and you can edit them. When writing and testing a robot, you use initial input variable values. When a robot runs in production, the input variables are initialized to values determined by the application running the robot. Note If the application does not provide values, the robot run will fail. Initial values for variables are the values that they have at the start of the robot (for example, at the first step). The values apply when you are writing, testing, and running the robot in production. Frames View The Frame view is located next to the Variables tab at the bottom right corner of Design Studio. 67 Kofax Kapow User's Guide The Frames tab shows all top-level browser frames and all their sub-frames in a tree. The view also contains a preview panel showing details about the selected frame. Starting from Design Studio version 9.6, the Frames view is the only place to get an overview of the frames. The labels of the top-level frames in the Frames tree are the same as the tab titles in the Page view. If the HTML page shown in the frame has a title, it is shown, otherwise the URL is shown. The labels of sub-frames are shown by their names (Unnamed (n) if they have no name). A node in the Frames tree may have various decorations such as: • An orange box around a label: the frame is the current window. • A gray box around a label: the frame is currently selected in the page view. • Light gray background color around the label: the frame is open in the page view. • The label and the icon is dimmed: the frame has no view (its viewport is zero height or zero width). The Frames Preview panel next to the Frames tree shows details about the selected in the tree frame. The details shown are the URL and a small rendering of the browser view of the frame with an overlay showing the size of the frame, for example 1263 x 1024. If a frame is blocked by URL blocking, then this is shown with both in the preview and in the tree. Note The Frames Preview panel is only available for robots designed with the Default browser engine. 68 Kofax Kapow User's Guide Frame View Actions There are a number of actions associated with the nodes of the frame tree. • Set as Current Window: inserts a "Set Current Window" step into the robot which is configured to open the frame with the name of the selected node (the name is shown in the tooltip on the node). • Close Window: inserts a step in the robot to close the frame • Open/Close: opens or closes a frame in the page view (a tab). Only works on non-toplevel frames since the top-level frames are always open. Note that this command does not insert any step into the robot. • Block URL: opens a dialog box for editing a URL blocking pattern for the frame and add this pattern to the robot's list of Blocked URL Patterns • Select in Browser View: selects the frame element in the browser view that defined the frame. If the frame containing the element is not open in the Page view, then this frame is opened. Note These actions are also available on the browser view tabs of the page view. Debug Mode The Robot Editor contains a specialized mode for debugging robots. Click Debug or Design on the toolbar to switch between design and debug modes. This is also available on the Design Studio Main Window toolbar. Alternatively, to debug from the current step in Design Studio, click Debug. The top of the Robot Editor in debug mode also contains a Robot view, similar to that of the design mode. Note The Robot View in debug mode has a current step only when you are actually debugging the robot. This current step is not always the same as the current step in the Robot View in the design mode. In the main panel, you see the results of the debugging process divided into various tabs. • Input/Output: List of all used variables and all values returned during debugging. • API Exceptions: List of API exceptions reported during debugging. • Log: The processing log generated during debugging. Some actions, particularly those that take a while to execute, such as the Loop Form action, write status information to this log. Step errors are also logged if configured to do so. • State: Whenever the debugging process is temporarily stopped, the State tab shows the robot state that is input to the current step. The State tab contains several sub-tabs. • Variables: Lists the variables. • Window, Cookies, and Authentication: Shows the state with associated dialogs. • Local Storage and Session Storage: Shows the HTML5 objects that have persisted locally. • API Exception: Generated at the current step. For all API Exceptions (and related errors), you can click the Goto button to navigate to the step that generated the error. The step that generated the error becomes the current step in Design Studio. • Summary: An overview of the number of variables returned or written to a database and generated API exceptions so far during the debugging process. • Stop When: Specify the criteria required to temporarily stop the debugging process. 69 Kofax Kapow User's Guide • Steps to Skip: Select steps to skip such as Store in Database, Delete from Database, Execute SQL, Execute Command Line, or Send Email. For details, see Debugging Robots. Type Editor Use the main window to configure the currently edited type. Among other things, you can configure the attributes of the type in the Attribute table. You can add new attributes, remove attributes, change their order, and configure attributes using the buttons below the Attribute table. Text Editor The Text Editor is a simple editor for plain text files (.txt) such as Readme files. The allowed extensions that the editor can open are .txt, .java, .jsp, .js, .log, .html, .xml, and .csv. The editor does not use the information that these extensions imply about file content. All files are treated as plain text files (no syntax highlighting). General Editing This topic gives a few general hints related to editing robots in Design Studio. These hints apply to when you make changes to a robot in the Step View, to a type in the Type Editor or to a text in the Text Editor. Copy, Paste, or Cut Use keyboard shortcuts to cut, copy, and paste items In Design Studio. • Ctrl-C Copy • Ctrl-V Paste • Ctrl-X Cut In addition, in most lists, such as the list of finders for a step, you can use Ctrl-Shift-C to copy all items in the list. Group and Ungroup Steps To group steps, select multiple steps and click Group on the toolbar. You can also right-click a step and select from the list. Some selections cannot be grouped. A group step must have exactly one ingoing connection and exactly one outgoing connection, and this must also hold for the selection of steps that you want to group. The only exception is when a selection of steps does not have any outgoing connection. In this case, you can group the selection, but the topmost End step must be connected to the end of the group. Take a look at the following example. 70 Kofax Kapow User's Guide In this robot, the following are examples of steps you can group: • All the steps • Any single action step alone, such as Step A, Step B, etc. • The branch point, step B, step C and the end step after step C The following are some examples of steps you cannot group: • the branch point and Step B (more than one outgoing connection) • steps B, C, D and the two End steps (more than one ingoing connection) You can select an expanded group step by either clicking (while holding down the Ctrl key) close to the connection or by including it in a drag selection. To ungroup a step or collection of steps, select the items to ungroup and click Ungroup on the toolbar, or from the context menu on the steps. Note The Group and Ungroup actions are inverse. If you group a selection of steps and immediately ungroup them again, the structure of the robot is unchanged. Use expand and collapse from the toolbar to perform the action on all groups. Use expand and collapse from the context menu to perform the action on the selected group or groups. The Expand and Collapse options on the context menu on steps do the same, but they are restricted to the Group steps in the selection. Drag and Drop In addition to actions, you can edit robot elements directly using drag and drop. As soon as you drag a step, special indicators appear showing valid drop locations. You can also select and move multiple steps at one time. • To move a connection endpoint, select the connection and move the mouse to one of the handles at the end. Next, click the handle and move it to a new location. As soon as you click a handle, special indicators appear, showing where you may connect it. • To abort a drag and drop action, move the mouse outside the robot and let go of the mouse button as shown in the following figure. 71 Kofax Kapow User's Guide Add a New Connection You can also create new connections using the mouse. Place the cursor near the end of a step so that an indicator appears (an orange circle with a green halo). Click the indicator and a new arrow appears. Keep the left mouse button pressed; move the mouse and a new connection will follow your mouse when you move it. New indicators appear and you can move the mouse to drop the new connection end point by releasing the left mouse button. Undo and Redo Changes While editing a robot, you can undo and redo every action. Click Similarly, click or select Ctrl-Z to undo an action. or select Ctrl-Y to redo an action. Step Validation As you edit your robot, the Robot View validates each step. Invalid steps are underlined in red. You can move the mouse to an invalid step to view an explanation of error. Types Write and maintain Types to define parameters used in a robot. It is important that you configure all of the relevant properties. Otherwise, the type may not perform as expected, or it may be invalid. Types must have: • A valid name. Type names must begin with a letter or an underscore and can only contain letters, digits, and underscores. Note In Design Studio, the name does not include an extension. For example, a type with the file name ExampleType.type is available in Design Studio as ExampleType. For more information about naming, see Naming policy. • A unique name. Two types in the same project must not have the same name. • A type kind which indicates how the type will be used. You can select the type kind using the "Type kind" drop-down list below the Attribute Table. Normally, it is not necessary to select anything as the type kind "Standard Type" is always used unless you need the legacy type kind "Database Output Type." See the reference documentation on Design Studio for more information. 72 Kofax Kapow User's Guide Type Attributes The attributes within a type must also be correctly added and configured in order for the type to be valid. You must specify both a name and a type for each attribute. The available attribute types are listed in the following table. Attribute Type Description Integer An integer, such as 12. The possible range is from -9223372036854775808 to 9223372036854775807, both inclusive. Number A number, such as 12.345. The possible range is from ±2.2×10−308 to ±1.8×10308 with slightly more than 15 digits of accuracy. Boolean A boolean value; either "true" or "false". Character A single character, such as "A". Short Text A short text. Displayed in a one-line text field. Long Text A long text. Displayed in a multi-line text box. Password A password. Displayed in a password field that shows asterisks instead of the characters in the password. HTML An HTML clip. This is the same as a Long Text, except that you can preview the clip in a browser window. XML An XML document. This is the same as a Long Text, except that only well-formed XML documents are allowed. Date A date, which must use the form yyyy-mm-dd hh:mm:ss.n, such as "1992-04-25 10:33:06.0". Binary Binary data; any sequence of bytes. Image An image. This is the same as Binary Data, except that you can preview the image. PDF A PDF document. This is the same as Binary Data, except that you can preview the PDF document. Session A session (containing cookies, authentications, etc.). Currency A currency code, as defined by the ISO-4217 standard, such as "EUR" for Euro. Country A country code, as defined by the ISO-3166 standard, such as "DE" for Germany. Language A language code, as defined by the ISO-639 standard, such as "de" for German. JSON A JSON value is either a JSON text or JSON Simple type where the JSON Simple type is either a JSON literal, a number, or a string. Step Actions and Data Converters In Design Studio, a short description is shown with each action and data converter. Click More next to the description to see additional information about the action or data converter associated with the description. You can also click help to get onscreen assistance associated with a selected step action or data converter. 73 Kofax Kapow User's Guide Several of the actions, such as Extract, can run extracted text through a list of data converters and sort the result in a variable. A data converter processes extracted text based on parameters you define. For example, the Extract Number data converter accepts an input text containing a number and outputs a text containing the same number in a standardized format. Because a data converter takes a text as input and outputs another text, data converters can be chained so that the output of one data converter becomes the input to the next data converter. The final output is the text output of the last data converter in the data converter list. For example, if the list of data converters contains the converter Convert to Upper Case, followed by a Remove Spaces data converter, the input text to the list is "R oboMa ker", is output as "ROBOMAKER". Patterns If you are new to patterns, we suggest you watch the introduction video on patterns. A pattern is a formal way of describing a text. For example, the text "32" can be described as a text containing two digits. However, other texts also contain two digits, such as "12" and "00" and can therefore also be described as a text containing two digits. We can express this by the pattern \d\d which is a formal way of expressing that a text must contain two and only two digits (\d is the symbol for a digit). We say that these texts match this pattern. Design Studio patterns follow the Perl5 syntax. A pattern is composed of normal characters and special symbols. Each special symbol carries its own special meaning. For example, the special symbol "." (dot) means any single character and matches all single characters, such as "a", "b", "1", "2", ... The table below provides an overview of the most commonly used special symbols. Special synbol Description . Any single character, such as "a", "1", "/", "?", ".", etc. \d Any decimal digit, such as "0", "1", ..., "9". \D Any non-digit, that is the same as ".", but excluding "0", "1", ..., "9". \s Any white space character, such as " " and line break. \S Any non-white space character, i.e. same as ".", but excluding white space (such as " " and line break). \w Any word (alphanumeric) character, such as "a", ..., "z", "A", ..., "Z", "0", ..., "9". \W Any non-word (alphanumeric) character, i.e. same as ".", but excluding "a", ..., "z", "A", ..., "Z", "0", ..., "9". Examples • The pattern ".an" matches all text lengths of three ending with "an", such as "can" and "man" but not "mcan". • The pattern "\d\d\s\d\d" matches all text lengths five starting with two digits followed by a white space and ending with two digits, such as "01 23" and "72 13" but not "01 2s". 74 Kofax Kapow User's Guide • If you want a special character, such as "." or "\", to act as a normal character, you can escape it by adding a "\" (backslash) in front of it. If you wish to match exactly the "." character, instead of any single character, you should write "\.". For example, the pattern "m\.n\\o" only matches the text "m.n\o". • You can organize a pattern into subpatterns by the use of parentheses: "(" and ")". For example, the pattern "abc" can be organized as "(a)(bc)". • All single characters are considered subpatterns. For example, in the pattern "abc", each single character "a", "b", and "c" is considered a subpattern. Subpatterns are useful when applying pattern operators. The following table provides an overview of the available pattern operators. Operator Description ? Matches the preceding subpattern, or the empty text. * Matches any number of repetitions of the preceding subpattern, or the empty text. + Matches one or more repetitions of the preceding subpattern. {m} Matches exactly m repetitions of the preceding subpattern. {m,n} Matches between m and n repetitions (inclusive) of the preceding subpattern. {m,} Matches m or more repetitions of the preceding subpattern. a|b Matches whatever the expression a would match, or whatever the expression b would match. Examples • ".*" matches any text, such as "Design Studio", "1213" and "" (the empty text) • "(abc)*" matches any number of repetitions of the text "abc", such as "", "abc", "abcabc", and "abcabcabc", but not "abca" • "(\d\d){1,2}" matches either two or four digits, such as "12" and "6789", but not "123" • "(good)?bye" matches "goodbye" and "bye" • "(good)|(bye)" matches "good" and "bye" As with other special characters, you can escape the special characters that appear in pattern operators by adding a "\" backslash in front of the character. Subpatterns are useful when you want to extract specific text pieces from a text. When you make a subpattern using parentheses, you can extract the part of the text that is matched by that subpattern. For example, consider the pattern "abc (.*) def (.*) ghi". This pattern has two subpatterns that are made by means of parentheses. If the pattern is matched against the text "abc 123 def 456 ghi", the first of those subpatterns will match the text "123", and the second subpattern will match the text "456". In an expression (see Expressions), you can refer to these subpattern matches by writing "$1" and "$2". For example, the expression "X" + $1 + "Y"+ $2 + "Z" will produce the result "X123Y456Z". This is a very important extraction technique in Design Studio. By default, the repetition pattern operators (*, +, {...}) will match as many repetitions of the preceding pattern as possible. You can put a "?" after the operator to turn it into an operator that matches as few repetitions as possible. For example, consider the pattern ".*(\d\d\d).*". If the pattern is matched against the text "abc 123 def 456 ghi", the subpattern "(\d\d\d)" will match the second number in the text ("456"), 75 Kofax Kapow User's Guide since the first *-operator will match as many repetitions as possible. If you put a "?" after the *-operator, so that the pattern becomes ".*?(\d\d\d).*", the subpattern "(\d\d\d)" will match the first number in the text ("123"), since the *?-operator will match as few repetitions as possible. We recommend that you experiment with patterns on your own. The best way to do this is to launch Design Studio and find a place where you can enter a pattern, such as in the Test Tag action. Then, click the Edit button to the right of the pattern field, to open the following Pattern Editor window. In the Pattern Editor you can enter a pattern and test whether it matches the test input text in the Input panel. When you open the window, Design Studio usually sets the test input text to the text that the pattern is matched against if the given step is executed on the current input robot state. However, you can also edit the test input text yourself, to try the pattern on other inputs. To test the pattern, click the Test button. The result of the matching appears in the Output panel. The Symbol button is very useful when you want to enter a special symbol in the pattern. When you click it, a menu is shown, from which you can select the symbol to insert in the pattern. This way, you don't have to memorize all the special symbols and their meanings. For more on the available special symbols and patterns, refer to documentation on Patterns. 76 Kofax Kapow User's Guide Expressions An expression typically evaluates to a text. For example, the expression "The author of the book " + Book.title + " is " + Book.author + "." evaluates to the text "The author of the book Gone with the Wind is Margaret Mitchell.", if the variables Book.title and Book.author contain the texts "Gone with the Wind" and "Margaret Mitchell", respectively. You can also do numeric calculations within the expression. For example, if the variable Book.price contains the price of a book, you can multiply it by 100 using the following expression: Book.price * 100 The following table provides an overview of the most commonly used sub-expression types. For a complete overview of all available sub-expression types, see the reference documentation on expressions. Commonly Used Sub-Expression Types Sub-Expression Type Notation Description Text Constant "text" or >>text<< Evaluates to the specified text, e.g. "Margaret Mitchell", or >>Margaret Mitchell<<. Variables variablename.attributename Evaluates to the value of the specified variable, e.g. "Book.author" might evaluate to "Margaret Mitchell". Current URL URL Evaluates to the URL of the current page. Subpattern Match $n Evaluates to the text matched by subpattern in an associated pattern (if any). For example, this is used in the Advanced Extract data converter, as shown below. $0 evaluates to the text matched by the entire pattern. Function func(args) Evaluates the specified function by passing it the specified arguments and converting its result to a text. Note that you can specify a text constant using either the quote notation or the >>text<< notation, for example "Margaret Mitchell" or >>Margaret Mitchell<<. If you use the quote notation, and you want a quote character to appear inside the text, you have to write it as two quote characters. For example, write "This is some ""quoted"" text" to get the text "This is some "quoted" text". If you use the >>text<< notation, anything can appear inside the text, except ">>" and "<<". Thus, you can write quotes directly, as in >>This is some "quoted" text<<. The >>text<< notation is useful for long texts that contain many quote characters, such as HTML. The following table shows the most commonly used functions in expressions. Function Description toLowerCase(arg) Converts the argument to lowercase. round(arg) Rounds the argument to the nearest integer. 77 Kofax Kapow User's Guide For example, the expression "The discount is " + round((Item.oldPrice - Item.newPrice) / Item.oldPrice) + "%." evaluates to "The discount is 10%." when the item's old price is $99.95 and the new price is $89.95. Experiment with Expressions We recommend that you experiment with expressions on your own. The best way to experiment with expressions is to launch Design Studio and open an existing robot. 1. In Design Studio, select the Extract action for the current step. 2. Add an Advanced Extract data converter. 3. Click the Configuration icon to configure the data converter. The Advanced Extract Configuration Window appears. 78 Kofax Kapow User's Guide In the shown example, note the use of the $n notation to extract parts of the input text. 4. Change the input text in the text area to the left. 5. Next, change the Pattern property. 6. Change the Output Expression property. Review the results in the right area, while typing the expression. Edit Expressions 1. On the Advanced Extract Configuration window, in the Output Expression field, click Edit. The Expression Editor appears. 79 Kofax Kapow User's Guide 2. In the Expression field, enter an expression, or click Expression to select one from the list. Options include Constant, Variables, Operators, Special Character, Functions, Page Properties, and Robot Properties with additional sub-expression functions. Expression values appear in the Input and Output sections. Note Testing functionality is not available everywhere in Design Studio. 3. Click OK. Projects and Libraries When working in Design Studio, you can have any number of projects open at any time. The purpose of a project is to develop a library containing a collection of robots and the files required by these robots. Typically, you create a project for each separate usage of robots, such as one project for each application in your company that uses robots. Two projects cannot share files; a type always belongs to one project, and the scope of a type is the project it belongs to. A project is a folder located anywhere in the file system. The project folder can have any name you want, but must contain the Library sub-folder. 80 Kofax Kapow User's Guide Note See Naming policy for more information about naming. Library This folder contains the library of the project. Place all robot files, type files, and other files used by the robots, such as files that are loaded from the robot library in the Library folder. You can organize the files in the Library folder using subfolders as appropriate. The following example shows a project folder named NewsAndStocksProject for a project that develops a robot library for extracting news from news sites and stock quotes from stock sites. NewsAndStocksProject/ Library/ News/ CNN.robot Reuters.robot News.type Stocks/ Nasdaq.robot NYSE.robot Stocks.type Note that this project has a Library folder with robot and type files divided into News and Stocks subfolders. When you close Design Studio, it remembers the projects and files open. The next time you open Design Studio, it will open the same projects and files. Current Project In Design Studio you can work with many projects, but the other applications in Kapow, such as RoboServer, always work on a specific project, referred to as the current project. When you install Kapow, a default project is created. This project is selected as current. If you open Design Studio the first time, this current project is the only opened project. If you close all projects before you close Design Studio, the next time you open Design Studio it opens the selected current project. You can change the current project selection using the Settings application, specifying the path to your new project folder in the Current Project Folder property in the Project tab, and then clicking OK to close Settings. Please see the Kofax Kapow Developer's Guide for more information. Shared Project Shared project is a project that is deployed on a Management Console and connected to a project on your local Design Studio computer. Management Console project can be shared between several Design Studios, thus several people can edit a project. When your shared project is out of sync with the project on the Management Console, the Shared Projects View visualizes the status of each object in the project. You can use different strategies when synchronizing your local copy with one deployed on the Management Console. Manipulate Robot Projects Use the following procedures to open, close, and create projects. • To open an existing project, on the File menu, select Open Project and select a project folder. The Open Project window appears. 81 Kofax Kapow User's Guide • To close a project, in the My Projects view, right-click the project. The Project window appears. Click Close. You can also close all projects from the File menu. • To create a new project, do the following: 1. On the File menu, select New Project. The New Project window appears. 2. Enter the name and location for the project. 3. Click Finish. A new project is created in the location you specified. The project folder name is the same name you assigned to the project. Example If you entered the name MyProject and the location: C:/KapowProjects then the following folders are created: C:/KapowProjects/MyProject C:/KapowProjects/MyProject/Library Organize Robot Files When you want to distribute and deploy your robot library in a runtime environment, such as RoboServer, you can pack the robot library into a single file called a robot library file. This will pack together all files contained in the robot library of the file in the current editor and save the result as a single file with a name that you assigned. Before creating the robot library file, save all open files, such as robots and types, to include the most current changes. You can make the robot library file available to RoboServer and execute robots from the robot library. See the Kofax Kapow Developer's Guide for more information. 1. In Design Studio, save all open project files such as robots and types. 2. On the Tools menu, select Create Robot Library File. The Select Robot Library Output File appears. 3. Navigate to the location to use for your library. Use the icons on the toolbar to change to Details or List view, move up one level, or create a new folder. 4. In the File Name field, enter a name for the library. 5. Click OK. The system creates the robot library file. 6. Click OK. Work with Shared Projects Once you connect to one or more Management Consoles, the Shared Projects view displays all projects deployed on all Management Consoles that you have access to. If projects and objects are not downloaded to your local computer, they are listed but not available. The Design Studio does not track such projects. 82 Kofax Kapow User's Guide Uploading a Project to a Management Console To upload a project to a Management Console, perform the following: 1. Right-click a project in the My Projects View and select Upload on the context menu; or select a project and click Uopload to Management Console on the Tools menu. 2. Select the Management Console and project that the files will be uploaded to in the Upload to Management Console window. Click Remember this (as a shared project) if you want to keep this project as shared and synchronize it between the Design Studio and the Management Console. 3. Click Upload to complete the procedure. After you upload a project to the Management Console, the project appears in the Admin > Projects tab and all project files appear in the Repository tab of the selected Management Console. Downloading a Project from a Management Console To download a project from a Management Console, perform the following: 1. Right-click a project in the Shared Projects View and select Download on the context menu; or select a project and click Download from Management Console on the Tools menu. 2. Select the name for the project and its location in the Select Project Name and Location window. 3. Click Finish to download the project. After you download a project from the Management Console, the project also appears in the My Projects View and you can edit the project files locally. Synchronizing Projects After you edit the files of the shared project on your computer, you can synchronize your local files with those deployed on the Management Console. Because a shared project can be accessed by several people, you might come across a synchronization conflict. The Design Studio provides messages and descriptions for you to understand what the conflict is, and how you can resolve it. Note that changed dependent files such as Types and Snippets can also prevent your robot from functioning properly. If you use Download to synchronize your project, the files are downloaded from the Management Console and your local changes are lost. If you use Upload, your local files are uploaded to the Management Console and any changes made by other people are lost (but those changes might as well be stored on their local computers). In any conflict situation, when changes made by you or other people can be lost, the Design Studio opens the Synchronize window for you to select the synchronization option. The following table provides synchronization examples. Status Synchronization Option Result The shared project files have been edited on your computer. No others who have access to the same project on the Management Console edited the files. Upload Your changes are uploaded to the shared project on the Management Console. If you select Synchronize, the default option is to upload your changes to the Management Console. The shared project files are changed on the Management Console. You know who edited the files and what the changes are. Download Changed files from the Management Console are downloaded to your local project. 83 Kofax Kapow User's Guide Status You edited the shared project files on your computer. Someone else edited the files and uploaded them to the Management Console while you were editing the same files. Synchronization Option Synchronize Result This is a conflict situation and you need to decide which changes to keep. In the Synchronize window you can select to either upload your changes to the Management Console, download the files from the Management Console, or just keep your files without synchronizing them with the Management Console. Interact with Databases You can use Design Studio to interact with databases. For details, see the following topics. • Map Databases • Types and Databases • Database Warnings • Create Database Tables • Store Data in Databases Map Databases Robots may need to access databases through various database accessing steps (such as Store In Database). You must provide a reference to a named database for these steps. The named databases used by a robot must be accessible from the RoboServers in order for the robot to be executed successfully on RoboServers. While designing robots in Design Studio, it is convenient to use local databases that are not available from the RoboServers. Rather than having to remember to change the named databases on the various database accessing steps before deploying a robot, Design Studio has an extra layer of abstraction to help overcome this problem: the database mapping. The mapping mechanism maps a named database in a database access step of a robot to a Design Studio database. As long as the robot is executed from within Design Studio, the named databases of the database accessing steps are mapped to the Design Studio databases specified by the mappings. The Design Studio user can use local databases while designing and testing robots without having to change the referenced named databases of the database accessing steps before deploying the robots. Using database mappings also makes it easy for the Design Studio user to create the robot store values in a different database: it is a matter of reconfiguring the mapping to make it point to a different database. A database mapping is a small configuration file defining which database to map to and whether Design Studio should display various warnings helping the user correctly configure the mapping and the referenced database. The name of the mapping is the file name of the configuration file. This means that if you create a mapping with the file name "objectdb," the database that the mapping points to will be accessible under the name "objectdb" in robots. Note that the databases may have the same names across different Management Consoles, to distinguish them when creating database mappings in Design Studio. A database name in the list includes a management console name as in the following example. 84 Kofax Kapow User's Guide The following steps show several ways to create a database mapping in Design Studio. 1. On the File menu, select New Database Mapping. A wizard appears. 2. Select a database and a project and click Next. 3. Enter a database mapping name and click Finish. When the wizard is finished, the mapping is created in the selected project and folder and is usable by robots. Database View 1. In the database view, right-click the database to associate with a project. 2. Select Add to Project and select the project to add the database to. 3. Enter the name to use for the database mapping. This is the mapping file name and the name the database will be accessible under. Notice that a name is suggested. This is the default database name, and the name used to access this database in other Kapow applications apart from Design Studio. 85 Kofax Kapow User's Guide Unmapped Database In Design Studio, when you open a robot using a database you do not have a mapping for, a warning is displayed. 1. Open a robot using an unmapped database. A warning appears, recommending a mapping with the name of the database referenced in the robot. This allows you to quickly run robots sent to you from developers who have other databases defined - without modifying the robots. 2. Complete the steps in the wizard. Types and Databases If your robot writes the values of variables to a database, the types of these variables need to define which attributes must be part of the key used to store the values in the database. The database key for the value is calculated as a secure hash of the attributes marked to be part of the database key. You can also specify a storage name as part of your attribute definition. This is an optional different name to use when storing the attribute. When saving values to database storage using the Store in Database action, the appropriate database table must exist in an available database. The table is required to contain columns matching the attributes of the type. See Creating and Deleting Database Tables for more information on how Design Studio can assist you in setting up the appropriate database tables. Consult Settings in Design Studio for more information on setting up database connections in Design Studio. Database Warnings Database warnings help you configure the database mappings, the robots and the referenced databases correctly. The warning system automatically monitors for potential problems such as type validation issues, missing tables, or missing database mappings and if an issue arises, a warning message is displayed in a status bar. 86 Kofax Kapow User's Guide Also, the warning system will monitor which databases names are used in robots and assist in creating any missing mappings. The system performs a shallow monitoring of the databases, which means that it does not constantly ping the databases to see if they are online, but rather updates the information as needed. The system caches the table structures of the relevant database tables and uses this cache to compute any warnings to prevent an excessive number of database queries. The cache is recreated when Design Studio knows something has happened that requires it. Any external modifications of the database tables or the database availability are not monitored. There is the option of rebuilding the database cache for a single database or for all databases. This option is available through the database view, and through and through warnings, as applicable. For instance, to recreate the cache for a database, right-click it in the Database view and select Refresh. Create Database Tables To store extracted variable values in a database, you create matching tables in the database. Design Studio can assist in creating these tables by examining the types you have created, and generating the appropriate SQL. When storing the value from a variable of some type, tables representing that type must be present in the database. 1. In the Tools menu, select Create Database Table . 2. 3. 4. 5. The Create Database Table window appears. Enter a name for your database. Define the database type, and table types that you want to create. Click Generate SQL. The system suggests a SQL statement for creating the tables. Modify, execute, or save the statement. 87 Kofax Kapow User's Guide The SQL shown is a recommended suggestion: you can change the statement to fit your needs, if required. For example, you might change the column type for a Short Text attribute from "VARCHAR(255)" to "VARCHAR(50)" to conserve database space, or you could add an autoincrementing primary key. However, under normal circumstances, you should not modify the table name or any column names, or remove any of the columns. Note If the database table already exists, it is dropped from the database when executing the SQL (because of the DROP TABLE statement at the beginning). Store Data in Databases This section explains how Kofax Kapow database storage works. Object Keys The tables you create for a type in a database have a column for each of the attributes in your type, plus an additional 7 household fields, named: ObjectKey, RobotName, ExecutionId, FirstExtracted, LastExtracted, ExtractedInLastRun, and LastUpdated. The most important field is ObjectKey, as it is the primary key for the table. Note The reason for the name "ObjectKey" is found in the terminology previously used in Kofax Kapow. Previously, types and variables were called "objects." To adhere to the new terminology, "ObjectKey" should be called "ValueKey." Renaming it would cause quite a lot of backward compatibility problems, though, and therefore it has been allowed to keep its old name. The ObjectKey for a type is what uniquely identifies values extracted from variables of that type when stored in a database. You have to figure out what uniquely identifies values of the type. If you are building a car repository, the VIN number may be enough to provide unique identification of each car. If you are collecting baseball results, you may need the year, team names, ballpark, and date to uniquely identify each match. As you build the type you can select how the ObjectKey is going to be calculated. This is done by checking the "part of database key" option when creating a new attribute. For our car example, the VIN number would be the only attribute marked as part of the database key, for the baseball match example, the attributes year, team names, ballpark, and date would all be marked as part of database key. The robot developer may also specify the key directly on the Store in Database action, to override the default algorithm defined on the type. The attributes that are not part of database key are sometimes referred to as non-key fields. For example, the car might have a price attribute, but even if the price changed we would still consider it the same car. Store in Database Kofax Kapow provides three actions for managing values in a database: Store in Database, Find in Database, Delete from Database. The Find and Delete actions are simple, but Store in Database does more than just store the value. Store in Database may insert a new value into the table, or update an existing value that was previously stored. Here is a list of exactly what happens. 1. When storing the value of some variable, the ObjectKey is calculated based on the variable's values of the attributes which in the variable's type are marked Part of Database Key. If the robot developer specifies a key on the action, this key is used instead. 2. Using the calculated key, a check is made to see if the value already exists in the database. 88 Kofax Kapow User's Guide 3. If the value does not exist, a new row is inserted into the database (under this ObjectKey). 4. If the value already exists, it is updated, and all the non-key attributes are written to the table (under this ObjectKey). Household fields Whenever a value is inserted all the 7 household fields are updated. On update, only some fields change. The following table provides an overview. Field Description Changed on ObjectKey The primary key for this value Insert RobotName The name of the robot that stored this Insert and Update value. ExecutionId The execution id for the robot execution that stored this value. Insert and Update FirstExtracted The first time the value was stored. Insert LastExtracted The last time the value was stored. Insert and Update LastUpdated The date when the value was last updated. Update ExtractedInLastRun If the value was extracted in the latest run (uses 'y' and 'n'). Insert and Update After each robot execution (in which the robot used Store in Database), all values previously collected by this robot, but not stored during this run, will have ExtractedInLastRun set to "n" and LastUpdated set to "now", indicating that the value was not found on the website during the latest run. Note If a value was found in the previous run, but no non-key fields have changed, then LastUpdated is not updated. However, if the value was not found in the previous run, but in a run prior to that, LastUpdated is updated even if the non-key fields have not changed. This means that the value was deleted from the site and then reappeared later. Harvest Tables The tables created by Kofax Kapow are often referred to as harvest tables, as the robots are harvesting data into them. To find out what information was available on a website the last time the robot was run, you can use the following SQL command: SELECT * FROM table WHERE ExtractedInLastRun = 'y' If you are running queries against a table at the same time as a robot is storing data into the table, the result is comprised of data from the previous run, mixed with whatever data the executing robot has stored so far. We recommend that you copy the data out of the harvest tables and into a different set of production tables, so you can run queries against a stable data set. There are many solutions where robots are used to store data in a database, but most of them fall under one of the three scenarios listed in the following table. 89 Kofax Kapow User's Guide Scenario Description Repository matching website (small data sets) The idea is to have a repository that matches the items on a website 1-to-1. Repository matching website (large data sets) Same as above, but the data set is too large to copy all data after every robot execution. Instead we want to update the production table after each robot execution, based on the changes that occur. The easiest way to accomplish this is have a truncated production table (deleting all rows) every time the robot is done executing, and then copy every record where ExtractedInLastRun='y' from the harvest table into this table. This works well for small data sets. This is where the LastUpdated field comes in handy. All values that have been updated have a LastUpdated field value larger than the start time of the robot. You can get the start time from the database logging tables, or you can have the robot store it somewhere. To detect deleted values, use the following command: SELECT * FROM table WHERE LastUpdated > 'StartTime' AND ExtractedInLastRun = 'n' To detect new values: SELECT * FROM table WHERE LastUpdated > 'StartTime' AND ExtractedInLastRun = 'y' AND FirstExtracted > 'StartTime' To detect updated values SELECT * FROM table WHERE LastUpdated > 'StartTime' AND ExtractedInLastRun = 'y' AND FirstExtracted < 'StartTime' Then update your Production table accordingly. Historic data The default setup allows you to see when a value was first extracted and when it was last updated, but you cannot see which run of the robot the value was found in. In this case, you should copy all the data from your harvest table into another table after the robot run, but in your new table the ObjectKey should not be a primary key. Instead, create an extra column called RUN_ID and use it together with the ObjectKey to create a compound primary key. If you don't need a RUN_ID you could simply create an auto-incremented column and use that as the primary key of your secondary table. Truncate the harvest table before each run. You don't have to copy all the household fields to your production table; only the ObjectKey is required for you to update your production tables Concurrency Considerations If you have multiple robots storing values of the same type to the same database, be aware of the following considerations. • Every time a value is stored, the RobotName column is updated. If you have two robots storing the same value (as identified by ObjectKey), only the last one will show after the robots are done executing. • If two robots store the same value at exactly the same time, you get an error. They both find that the value is not in the table and try to insert it, but only one of them will succeed. In most cases, the error can be ignored because it is the same value. • If you run the same robot twice at the same time and the robot stores data in a database, you break the way the ExtractedInLastRun column is used. When the first robot is done executing, it updates the ExtractedInLastRun to "n" for all values it has not stored. This includes all values stored by the second robot so far. Later when the second robot finishes, it sets ExtractedInLastRun to "n" for all values stored by the first robot, completely negating the first run. 90 Kofax Kapow User's Guide Value Relations The storage system does not provide an automated way of managing relations between values. If you have a value of type Person and one of type Address, and you want to link them, you have to maintain this link. The easiest way to create a link is to have the ObjectKey of the Person value be a foreign key in the Address value that should be linked to this person. If the ObjectKey is calculated automatically from the type, you can use the Calculate ObjectKey action to generate the key and assign it to each of the address values before you store them. You should be careful when building robots with connections between stored values. If an error occurs when you store the Person value, make sure that no Address values are stored. ObjetKey Caveats If you are using MySQL, Oracle or Sybase, review these important ObjectKeys rules. • On Oracle empty string is stored as null. • On Sybase, an empty string is stored as " " (a string with a single space). • MySQL does not have millisecond precision in timestamps. These three cases all result in a potential loss of data when the data is stored in the database. The ObjectKey is calculated inside the robot based on the data in the given variable. If you later load the value from the database and try to recalculate the ObjectKey, the ObjectKey is different if data loss occurred in any of the attributes marked as part of database key. Robot Structure Robots mimic human behavior; they do (more or less) what you do when you are looking for content on the Internet using a browser: You start by searching for the content. Once found, you read and process it. Similarly, most robots can be divided into two parts: a navigation part and an extraction part. Navigation is concerned with "getting to where the content is." Navigation mainly includes loading pages and submitting forms. When navigating in Design Studio, you typically use the Click action to navigate through and among web pages. Extraction is concerned with "getting the right content." Extraction mainly includes selecting, copying, and normalizing content from a web page that you navigate to. When extracting in Design Studio, you typically use the Test Tag action to skip uninteresting ("noisy") content, the Extract action to copy content into variables, and the data converters for normalizing the content so that it gets the format you want, such as the right date and number format. Once extracted, you output the value with the Store in Database or Return Value action. A typical robot starts with one or more steps, each containing a Load Page or Click action to navigate to the interesting content on a web site. It proceeds with one or more steps, each containing an Extract action, and ends with a step storing or returning the extracted value. Note that in many robots the navigation and extraction parts overlap because the content to extract is located on several pages. Again, this is similar to looking for content yourself; often, you have to visit several pages to get the content you want. 91 Kofax Kapow User's Guide Most robots include other actions than the ones mentioned above, such as a For Each Tag action for loading several similar looking pages or extracting values from several similar looking table rows. Because robots have different tasks, they have different needs. For this reason, we have included a considerable number of step actions and data converters in Design Studio. Start by familiarizing yourself with the basic and most commonly used step actions and data converters, and then begin to explore. Experience shows that you can create most robots using only a handful of step actions and data converters. So, find your own favorite step actions and data converters and stick to them until you feel a need to explore others. Write Well-Structured Robots Writing well-structured robots is essential because each robot is a program. Writing unstructured robots is like writing books with no chapters or table of contents. Writing well-structured robots is important because: • It helps document the robots. • It makes it easier to maintain the robots. • It makes it easier to find your way around the robots. A side-effect of writing well-structured robots is that is that it can also make them load faster in Design Studio. As a result, robots are generally more responsive when they are edited in Robot View. The two main tools for writing well-structured robots are Snippet steps and Group steps. Both step types are a way to take a part of a robot, give it a descriptive name, and pack it up in a single step. This way you can forget what the part of the robot does in detail and concentrate on the overall structure of the robot. This concept is similar to those in other programming languages, such as methods, functions, and procedures. You use a group step to pack up and hide steps that perform a well-defined task. Give the step a descriptive name, such as Login to site X, Report error. It is important to give a relatively short descriptive name to the group step that describes what the steps inside the group do. If you cannot provide a good name, then it may be because the group does not perform a well-defined task. By introducing a group step you help document your robot, because the name describes what this part of the robot does. Although snippets are mainly introduced to share functionality between robots, they can also be used inside a single robot to help structure it. If you have a collection of steps in a robot used in several branches, such as connections from different parts of the robot joining at the start of the steps, you can replace such steps sharing by introducing a snippet containing the steps. The following robot structure uses snippets and groups instead of joining connections. The last two steps c and d are shared by the two branches starting with the steps a and c. In real life you probably have a much larger robot and more than two branches sharing steps in this way, and the steps 92 Kofax Kapow User's Guide involved may be far apart. As a result, it may be difficult to get an overview of the robot. As a first step towards getting a better structured robot, you can introduce a snippet step containing the steps c and d as follows. You can edit the steps inside the snippet steps and still be sure that the changes are shared in the two branches. You can structure the robot further by putting both branches into a group step: Finally, you can use the two group steps and get the following simple robot. This resulting robot does two tasks, one performed by Group1 and the other performed by Group2. By giving these two groups descriptive names, the robot has a more logical structure than the original robot. Admittedly this is a very simple example, but when robots get beyond a certain size and contain connections crisscrossing the Robot View they can become overly complex. Restructuring the robot in the manner described above may help ensure that the robot overview is manageable. 93 Kofax Kapow User's Guide Determine the Page Type You can create a Try step to identify the type of loaded page. Valid page types include HTML, XML, Excel, and Binary. 1. 2. 3. 4. 5. 6. 7. 8. In the Designer, after a Load Page step, insert a Try step. On the branch, Insert an action step and select Test > Test Page Type. On the Action tab, Page Type field, select HTML. On the Basic tab, change the Step Name to Is HTML Page. Select the try step and click Add Branch. Repeat 2 through 5, with Page Type set to XML, and the Step Name to Is XML Page. Repeat 2 through 5, with Page Type set to Excel, and the Step Name to Is Excel Page. Repeat 2 through 4, with Page Type set to Binary, and the Step Name to Is Binary Page. Once the robot runs, the page type is highlighted. Use Tag Finders Use Tag Finders to find a tag on an HTML/XML page. The most common use of a Tag Finder is in a step, where the Tag Finder locates the tag to which you want to apply an action. The list of Tag Finders for the current step is located in the Tag Finders tab of the Step View. Tag Paths A tag path is a compact text representation of where a tag is located on a page. Consider this tag path: html.body.div.a This tag path refers to an -tag inside a -tag inside a -tag inside an -tag. 94 Kofax Kapow User's Guide A tag path can match more than one tag on the same page. For example, the tag path above will match all of the -tags on this page, except the third one: 1 2 3 4 5 6 You can use indexes to refer to specific tags among tags of the same type at that level. Consider this tag path: html.body.div[1].a[0] This tag path refers to the first -tag in the second-tag in a -tag inside an -tag. So, on the page above, this tag path would only match the "Link 4" -tag. Note that indexes in tag paths start from 0. If no index is specified for a given tag on a tag path, the path matches any tag of that type at that level, as we saw in the first tag path above. If the index is negative, the matching tags are counted backwards starting with the last matching tag which corresponds to index -1. Consider this tag path: html.body.div[-1].a[-2] This tag path refers to the second-to-last -tag in the last-tag in a -tag inside an tag. So, on the page above, this tag path would only match the "Link 5" -tag. You can use an asterisk ('*') to mean any number of tags of any type. Consider this tag path: html.*.table.*.a This tag path refers to an -tag located anywhere inside a