Conditional workflows

In EasyMorph, a conditional workflow is one in which a part or all of the workflow may be executed or skipped depending on a condition. For instance: IF Table 1 is empty THEN skip deduplicating columns.

Watch a similar video in

Conditions can be applied in EasyMorph is many ways, and many actions have conditional options within their configuration. There are however several actions which are specifically designed for adding conditional behaviour to a workflow. When placed in a workflow, they can affect the execution of other actions, or the entire workflow based upon a defined condition.

These conditional actions can be split into 2 main types based upon the affect they have:

  • Skip actions are those which can cause one or more following actions to be skipped.
  • Halt actions are those which can stop the entire workflow from executing.

Skip Actions

Skip actions skip the execution of all other actions which follow them if a specific condition is met. Their action is limited to the table in which they are placed; they do not skip any actions in other tables. Some common use cases for skip actions are:

  • To prevent performing other actions if they would be unnecessary because the table is, or is not, empty.
  • To skip loading in historical data during an incremental data load if a full reload is required (e.g. when running overnight).

There are currently two skip actions in EasyMorph which have very similar behaviour and configuration.

Skip actions on condition

The first is the Skip actions on condition action which makes EasyMorph skip all the actions that follow it, when the action’s condition is met. The condition is applied to the result of the action before it in the table and can be one of four main types of conditions as can be seen below.

Skip actions on condition

The first 2 types are self-explanatory and skip based upon the table being empty or not. The other types, when selected, provide the ability to add an expression which must equate to true or false. This expression can be used to define almost any desired condition.

Whatever condition type is selected, if the condition is not met (i.e. is false), then the following actions will continue to be executed by EasyMorph. As can be seen below, their icons remain unchanged.

Actions are not skipped

If the condition is met (i.e. is true), then the following actions will be skipped and will not be executed by EasyMorph. As can be seen below, their icons have an additional red skip icon overlayed to clearly indicated they have been skipped. Note that the skip action’s icon has also changed to indicate visually the negative evaluation of its condition.

Actions are skipped

Skip on condition in another table

The second skip action is the Skip on condition in another table action which has all of the same configuration options as above. It differs from the "Skip actions on condition" action in that instead of applying its condition on the result of the action prior to it, a specific table can be named upon which the condition should be applied. Skip on condition in another table

The selected table can be any table from the module within which the action is place. Like above, the icon of the "Skip on condition in another table" action and those of actions following it, will change to indicate if the condition is not met and skipping is occurring.

Halt Actions

Rather than skipping, sometimes we may wish to halt the execution of the entire workflow if a condition is met. Some common reasons to use halt actions would be:

  • To check input data for values out of the acceptable range, missing or extra values which may prevent later workflow actions from having their desired effect.
  • To verify data consistency after a merge action.
  • To prevent erroneous data from propagating into other systems or databases later in a workflow.

EasyMorph currently provides two such halt actions.

Halt/warn on condition

The first is the Halt/warn on condition action which is quite similar to the skip actions we've looked at above. The first option is whether we wish to halt the execution of the workflow, or if instead we wish to give a warning but to continue the workflow execution.

Halt/Warn on condition

If halt is chosen, when the defined condition is met (i.e. equates to true) then the execution of the entire worflow is stopped at that point and no further actions in any table will be performed. The halt action's icon also changes to include a yellow triange to indicated the condition was met and all subsequent actions, whether in the same table as the halt action or any other table, have a green "run action" icon overlayed to show visually they have not been performed.

Halt/Warn on condition

If warning is selected, then when the condition is met, a warning is given but the execution of the workflow continues. This can be useful for alerting of none critical issues which shouldn't affect the rest of the workflow. As can been seen above, there are the same 4 types of conditions for us to choose from as there are for the skip actions. An optional error message can also be configured.

Halt on data type mismatch

The second halt action is Halt on data type mismatch. This action has different configuration options compared to the other halt and skip actions. It allows for the workflow execution to me stopped if the table of data resulting from the previous action does or does not contain specific data types.

Halt on data type mismatch

Firstly, we configure if the specified data type is to be considered as desired (i.e. we expect this data type only) or is not desired (i.e. we do not expect to see this data type, but all other data types are OK). Worth noting is the "error" data type which is a special data type and is analogous to when Excel shows "#DIV/0!" in a cell when calculation attempts to divide by zero. And finally, we can decide whether to apply the data type check to the the entire table or to specific columns that you can select.

One common question from those new to EasyMorph is how to configure different data type checks on different columns. Whilst this can't be done in a single "Halt on data type mismatch", there is nothing preventing you from adding multiple instances of the action to the same table.

Advanced topics

Conditionally derived tables

The "Skip actions on condition" action described above handles simple cases when a group of actions is either executed or not, depending on a condition. However, sometimes there can be two (or more) groups of actions executed on mutually exclusive conditions. For instance: IF loaded data has no missing dates THEN export it into a database ELSE send an email to Pete. In such a workflow, only one thing can happen during workflow execution: either data is exported into a database, or an email is sent to Pete. Both condition branches are mutually exclusive and must never be executed in the same workflow run. If we depict this workflow graphically, it would look as below:

Conditional workflow animation

Such conditional branching is arranged in EasyMorph using conditionally derived tables. To remind you, a regular derived table is a table which replicates the resulting state (data) of its source table and performs certain actions with it. If you're not familiar with derived tables, check out this tutorial article: Derived tables.

A conditionally derived table only executes its actions when a certain condition is satisfied. If not, then no data is replicated from its source table, and no actions are performed — all actions in such table are skipped. The example mentioned above would look slightly different if we do branching using conditionally derived tables in EasyMorph:

Conditional workflow using derived tables

In this example, we have 2 conditionally derived tables. The conditions in the tables are inverse (i.e. mutually exclusive) to each other: in one table the condition is "missing a date?" while in the other it's "NOT missing a date?". Therefore, depending on whether the loaded data is incomplete or not, either one table or the other is entirely skipped.

In an EasyMorph project, the example described above would look as follows:

Conditional workflow using derived tables

The two rightmost derived tables are derived conditionally. Notice that one of the tables is skipped — you can see it has different title bar color, and a derivation icon with red "X". In this table, actions were not executed and it has no data.

The project depicted above detects if some dates are missing or not by calculating the difference between adjacent dates. If no dates are skipped then the difference from previous date will always be equal to 1, for all dates. If one or more dates are skipped somewhere, then for the following date the distance will be 2 or more. We calculate the max distance, and merge it into the main table as a new column using the "Peek" action. Later, prior to exporting, the column is removed.

In the conditionally derived tables, the conditions are as follows:

[Distance, days] = 1

for the upper derived table "Export to DB", and

[Distance, days] <> 1

for the lower derived table "Send an email".

As you can see, the conditions are inverse to each other. Therefore, either one or the other derived table is skipped, but never are both skipped or executed simultaneously.

Conditions

The "Derive table" action has a switch that tells it whether it should be unconditional or conditional. In the latter case, there are three possible conditions for derivation:

  • If the given expression evaluates to TRUE for each row in table
  • If the source table is empty
  • If the source table is NOT empty

In the screenshot below, you can see the properties of the "Derive table" action in the conditional mode:

Conditional workflow

Example

Download the example from this Community topic: If a file doesn't exist.

More complex workflows

The examples above show using conditionally derived tables for the "IF...THEN...ELSE" type of workflows. However, in the same fashion, it is possible to arrange the "SWITCH...CASE" type of branching that arranges 3 or more branches.

Also, if a condition branch requires executing a complex workflow that can't be fit into one table, the workflow can be moved into a separate module and executed using the "Call" action (explained earlier) in the respective condition branch.