Usefulness5/5
Frequency: 5/5
Power: 3/5
Unintuitiveness: 3/5
Complexity: 3/5

Queues

Connect different sub-processes with a queue

last changed at 2021-01-11 18:49:34 (UTC)
Intermediate
Process Design
Splitting Things Up
UiPath

Splitting Things Up

This pattern is part of a series showcasing different techniques to split your business process into smaller pieces. Make sure to check out Splitting Things Up for an overview.

Situation

You developed a workflow and split it up into sub-processes. Now you need to exchange some transactional data between the sub-processes or prioritize some transactions over others - but how?

Compare

It is highly recommended to read Sub-processes before going into queues. This will give you an understanding of how to split up a single process into multiple parts.

Common examples

  • A process involves many similar, but independent, items (called transactions or queue items)
  • Preparing the environment takes a long time, but the actual transactions are executed quickly
  • New items have to be handled as quickly as possible (by keeping an unattended robot waiting for them)
  • One part of a process takes a lot more time than another (load balancing)
  • You need a log of input data the robot ran on
  • To create an interface for external applications that feed data into a robot
  • To prioritize certain transactions over others
  • SLAs apply to individual transactions

Solution: queues

Queues are just what they sound like: a (mostly) first-in-first-out list of independent items you need handled. Queue are created in Orchestrator, which will handle the pesky assignment to the different robots (and prevent race conditions) for you.

Queue basics

When thinking about queues, I generally picture a conveyor belt with robots at either side: Robots on the producer side put some process information into packages (called queue items or transactions) and then place them on the conveyor. Robots on the consumer end take the packages from the conveyor and act based on the information within.

Queues are great in many ways: they make reasoning about the process easier, give you a natural point to inspect for potential problems, allow for some automatic load balancing, and more. Therefore, they should definitely be part of every RPA developer’s toolkit.

Perhaps the most important non-obvious difference to sub-processes is that queues are a declarative interface rather than imperative instructions. That is, the producer only specifies that it would like a transaction to be handled, but it doesn’t need to tell Orchestrator exactly how to do it.

On the other side of the conveyor, the consumer doesn’t care where the queue items come from — only their properties matter. This allows you to transparently replace either side of the equation with a different implementation, or use multiple different producers for the same queue (e.g. a supplier creation might be triggered on demand by a human via an attended robot or automatically by an invoice handling robot).Multiple consumers are also possible but less common because there is typically only one best way to handle an item at any given point in time. One useful application is to keep an old and a new implementation running in parallel to de-risk a transition phase.

Case study

I think this pattern is best learnt by example, so we will continue our exercise where we left it in sub-processes. To refresh your memory: we are implementing a supplier master data creation/change process targeting an example web app from UiPath (read all about it here).

Previously, we split the process into two parts, which we called manager and worker. The manager corresponds to the producer side of the queue and the worker to the consumer side.**

We will reuse both of these processes, modifying the manager to create queue items instead of starting the worker process, and creating a wrapper for the worker that uses the RE Framework to handle the queue boilerplate.

Server stuff

We have to create some stuff in Orchestrator… Boring! Alright, I guess there’s no way to wiggle out of this one, so we might as well get it over with. Go to your favorite Orchestrator tenant (mine is affectionately called sreutterDefault), choose a folder, and go to the Queues menu. If you’ve ever wondered what that is for — now you know.

The Queues view
The Queues view

Click the nice big ”+” on the right-hand side to add a new queue. Let’s call it Supply_Drama_Q. No, you may not call it anything else. This is the only allowed name and using a different one will result in termination. Make sure you disable the Auto Retry feature, which is enabled by default, but can be a pain during development. Unfortunately, it cannot be re-enabled later, but since you should be using a different Orchestrator or tenant the production deployment anyways, this is usually not a big concern.

Create queue
Create queue

Alright, that’s enough about Orchestrator for now. There are quite a few more settings you could make, but we’ll limit ourselves to the basics here.

Manager of Queues

The first step is to make a copy of the manager project from sub-processes, rename it and modify the project settings accordingly.

Changing the workflow is quite simple: all we have to do is to open EnterSupplierChanges and replace the Invoke Process call with an Add Queue Item. I merely commented out the Invoke Process because we’ll need a very similar thing later.

Add Queue Item
Add Queue Item

Unfortunately, Studio doesn’t allow us to copy&paste arguments in the collection editor. There is a trick to help, which is, however, not for the faint of heart:

  1. Add a single argument to Add Queue Item (so you don’t have to manually create the tag)
  2. Close EnterSupplierChanges in Studio
  3. Make a backup just to be safe
  4. Open EnterSupplierChanges.xaml in your favorite text editorGood text editors for Windows include, but are not limited to, VS Code, Notepad++, Sublime Text and Atom. Windows Notepad and Word are disqualified
  5. Copy the arguments from Invoke Code to Add Queue Item (finding them reliably can be hard if you’re not good at reading XML)
  6. Save and re-open in Studio. This trick will probably get its own pattern at some point, in a future guide with a few other XAML editing tricks that can save you hours of repetitive work
Copy & pasting arguments in Notepad++
Copy & pasting arguments in Notepad++

Assuming you didn’t screw anything up with your XAML hacking, you should end up with something like the following.Don’t say I didn’t warn you! Also remember to set the queue name property (and the Orchestrator folder if you don’t plan to keep the queue in the same folder as the process).

Add Queue item beautifully configured via XAML hacks
Add Queue item beautifully configured via XAML hacks

Et voilà, our work on manager is done. Test it to see if you get the expected two queue items when dropping our test file in the hot folder. Feel free to publish to Orchestrator, but it’s not mandatory for manager

Queue item created by the new manager
Queue item created by the new manager

Stand in Line, Workers!

Now to tackle the worker. Because of where we previously split our process, we don’t actually have to touch our worker implementation - it already performs only a single supplier creation or change, based on simple value process arguments.

Our job is, instead, to create a new project based on the Robotic Enterprise Framework, get the data from the queue item, and use Invoke Process to run our already-published worker. I will assume that you already know about the RE Framework - if not, this is a good time to visit the UiPath Academy to learn about it.

Creating an RE Framework project
Creating an RE Framework project

Worker configuration

First, we need to tell the process where to find the queue items it should feast on. In RE Framework, this is done in Data/config.xlsx. While we’re at it, we might as well change the business process name to make the logs prettier. We don’t need to change the other config parameters for our purposes here.Personally, I prefer to store my config in text files rather than Excel but that is beyond the scope of this article

Config settings for RE Framework
Config settings for RE Framework

Process changes

The main changes we have to do are in Process.xaml, where we have to get our queue item information and use Invoke Process.

One way to push the data into Process is to export all of our 12 arguments as an out argument in GetTransactionData and as an input argument in Process. I’m usually too lazy to do that,With the exception of id fields and similar data for error messages so let’s just get the variables from the queue item from within Process.xaml.

The quickest way by far to set this up is to copy the variables, variable assignment and the commented-out Invoke Process from our manager workflow.Told you we’d need it

Process.xaml after copying stuff into it
Process.xaml after copying stuff into it

We have the misfortune of having to replace all of the assignments with their respective queue item versions. Cough… XAML hacking… cough. to mix things up a little, I used the following replacement in VS Code to make my life easier, then manually fixed the remaining small naming differences.

Find and replace in VS Code
Find and replace in VS Code

When testing this workflow, I also found that the current date expression doesn’t work for SpecificContent in quite the same way it does for DataTables, so replace it with this one:

if(not in_TransactionItem.SpecificContent("CreationDate") is Nothing, Convert.ToDateTime(in_TransactionItem.SpecificContent("CreationDate")).ToString("MM/dd/yyyy"), Nothing)

Init and Close applications

Last, we can also add some Use Browser activities so we only open and close the site once instead of once per transaction. These go into Framework/InitAllApplications.xaml and Framework/CloseAllApplications.xaml respectively.

Init and close applications
Init and close applications

Final result

Between our good preparation, XAML hacks, and much of the heavy lifting being done by the RE Framework, this is all we have to do to introduce a queue between our two process parts. Pretty quick work, wasn’t it?I’d estimate that I’d normally have been done in less than 1 hour, but don’t have an exact measure because I develop the example workflows and write the articles in an interleaved manner

As always, you can find my finished example projects in this guide’s github repository. Worker wrapper and manager

Variants

  • Add Queue Item And Get Reference is the long-running workflows version of Add Queue Item and allows the parent process to wait until the queue item has been processed. This can be useful if you want to chain several queue-based processes together.Just keep in mind LRWFs don’t work on attended robots
  • Bulk Add Queue Items lets you add a whole data table of queue items at once. It would have worked just as well for our case study - so try refactoring to it as an exercise.
  • Wait Queue Item is very similar to Get Transaction Item, the difference being that, if no queue item exists, the workflow will wait at this activity until a transaction arrives (or a specified timeout is reached).
    This blocks a robot, but allows you to almost instantaneously process the first queue item that arrives. This can be a worthwhile tradeoff when a quick reaction time is paramount and transactions trickle in independently, such as in live customer-facing situations.There are more advanced patterns involving this, such as using a single queue for multiple such processes, but these will have to wait for another day

Name confusion

Unfortunately, the UiPath dev team seems to have gone out of their way to make the names for the queue-related activities as confusing as possible. I would like to take this opportunity to clarify which of these you should use.

First, don’t confuse Add Queue Item with the practically useless Add Transaction Item. The latter creates a queue item already “in progress” (from within the queue consumer). I guess you can use it to create a record of what the consumer works on (e.g. to analyze in Insights or Kibana later on), but have not seen it applied anywhere so far.

Second, to make things even more confusing, in the consumer you want Get Transaction Item or Wait Queue Item, but not Get Queue Items. This is because Get Queue Items does not set a queue item to In Progress and, therefore, two robots could in simultaneously try to pick it up - which leads to every concurrent system developer’s nemesis: a race condition.There are legitimate uses for Get Queue Items, such as in reporting processes, however

TL;DR: the producer uses Add Queue Item or Bulk Add Queue Items, the consumer uses Get Transaction Item or Wait Queue Item

Comments

  • Queue triggers: since the 19.10 enterprise release, you can ask Orchestrator to automatically spawn a consumer process (in our case the worker wrapper) when new queue items are added. I recommend this as the default way of running them unless you have a specific reason to go for a different solution.
  • If you go for a schedule rather than a queue trigger (e.g. when using Wait Queue Item), you’d usually run the queue consumer pretty frequently to check for new queue items. How often depends on your SLAs, but between 1 minute and 30 minutes seem to be typical values. When doing this, Get Queue Items can be gainfully employed to create a fast path that exits early — without initializing the environment — if no queue items are available.
  • If you don’t use RE Framework, don’t forget to Set Transaction Status after finishing a queue item. It is very easy to forget this and end up with orphaned in progress items.
  • While debugging the consumer, don’t use the stop button if errors occur. Make sure it goes into the error handler and marks the queue item as failed first. Otherwise the queue item will get stuck in In Progress and you’ll either have to wait 24 hours or create a workflow to unstick it.
  • There are many advanced settings for queues we will not cover here, but make sure you familiarize yourself with them as they can be very useful depending on your application
  • Because the producer doesn’t need to know any details about the consumer - only about the queue interface - queues naturally lead to greater separation between the two. For example, it would be easy to split the worker into multiple processes, or replace its implementation entirely, without having to worry about repercussions on the producer.

Exercises

  1. Refactor the case study to use Bulk Add Queue Items
  2. Modify the worker wrapper to use Wait Queue Item
  3. Create a queue trigger to start the worker wrapper automatically when ueue items are added
  4. Why did we not have to modify the worker process at all?
  5. Familiarize yourself with references, auto-retry, SLAs, priority, the postpone setting and queue schemas

Solutions

  1. Because it already works on simple data types passed in as progress arguments, we could opt to write a wrapper around it instead.

References


© 2021, Stefan Reutter