Loops and iterations

Iterations in EasyMorph are used to arrange loops (cycles) — i.e. when part of a calculation is executed multiple times, each time with a new set of input parameters (data). Using iterations you can transform multiple files in a similar fashion, process daily data for a given range of dates, send emails to multiple recipients, and arrange other repetitive workflows. Understanding iterations may require a bit of effort. However, once learned, iterations become a very powerful technique that can be used in many real-life scenarios. From a programming perspective, iterations in EasyMorph allow arranging FOR...EACH, FOR..NEXT, and DO..WHILE/UNTIL types of loops.

This article requires a good understanding of parameters and modules/subprojects explained earlier in this tutorial.

In EasyMorph, iterating means calling (executing) another module (or project) multiple times. In this chapter, we will be talking mainly about iterating modules. However, it all is equally applicable to iterating projects.

The "Iterate" action

FOR..EACH type of loops in EasyMorph are arranged using the "Iterate" action. The action is very similar to the "Call" action (described in the previous chapter) with two major differences:

1. Parameters of the called module can be assigned using column values.
2. The called module is called not once, but as many times as there are rows in the table with the "Iterate" action.

The diagram below shows how Module A iterates Module B using a list of file names. Module B has a parameter named Param1. This parameter is assigned by Module A with a new file name every time it runs Module B. Because the input dataset (Table 1) in Module A has 3 rows, the "Iterate" action calls (iterates) Module B exactly 3 times — once per each file name in column [File Name].

Frequently, iterations are performed against a list of files, or a list of dates. Such lists can be generated using the "Calendar" action (generates a sequence of dates) or "List of files" (generates a list of files in a folder). Alternatively, they can be imported from a database or a text file.

The "Iterate" action runs another module (or project) once for each row of the action's input dataset. Parameters of the iterated module can be assigned using field values of the dataset. If a module parameter is not assigned, it keeps the last saved value (the default value).

The "Iterate" action can run in one of the two modes:

• Iterate — do not return anything from the iterated module, i.e. just run it one or several times.
• Iterate and append results — run the module one or several times, return a table (the result dataset) from each iteration, and append all result datasets into one, which becomes the output of the "Iterate" action. Appending the result datasets is similar to the "Append" action — columns with the same name are concatenated into one column, columns with unique names are extended with empty cells.

To create a FOR..NEXT type of loop, generate a sequence of numbers from 1 to N using the "Generate sequence" action, then iterate across the sequence using the "Iterate" action.

Iterating modules

All calling actions ("Call", "Iterate", "Iterate table", and "Repeat") can run modules in the current project as well as external projects. When designing iteration of a module, it may be confusing to see that the module's parameters that are visible in EasyMorph don't change after a call from another module. That's because during calls/iterations, under the hood, EasyMorph creates an invisible clone of the iterated module and assigns parameters to that invisible clone and runs it. The module that you see remains intact, calls don’t modify it. This is done in order to avoid ambiguity and conflicts because the same module can be called from multiple places simultaneously.

In order to understand how a module would be calculated when called/iterated from another module, set the called module's parameters manually and run it (if Auto-run is off).

See iterations explained

Examples

We've prepared a pack of 5 iteration examples. Each example has an illustrated description and annotations in project tables and actions.

• Example 1 – load multiple files
• Example 2 – split one file into many files
• Example 3 – batch processing of several files
• Example 4 – iterate and append data into one table
• Example 5 – iterate another table

Advanced topics

The "Iterate table" action

The "Iterate table" action is basically the same as the "Iterate" action with the only addition being that, besides assigning parameters of the iterated module, it also allows passing a table into it. The iterated module should use the "Input" action in order to obtain the table. This action is helpful when multiple calculations with different parameters have to be performed on the same dataset (e.g. financial stress testing).

Using the "Iterate table" action allows arranging two more iteration scenarios:

The "Repeat" action

Loops created using the actions "Iterate" or "Iterate table" are used in cases when the number of iterations is known in advance and depends on a condition. However, there are cases when the number of iterations is not known ahead of time and depends on a condition. For instance, a web request that fetches paged data from a web API needs to be repeated again and again until no more data is fetched. In programming, such cases are handled using a DO...WHILE/UNTIL loop. In EasyMorph, such loops are arranged using the "Repeat" action.

The "Repeat" action has two modes:

• Run another module/project UNTIL its result table becomes empty
• Run another module/project WHILE its result is empty

In the "Repeat UNTIL" mode, the action can automatically concatenate all non-empty results into one table (similarly to "Iterate"), or just return the last non-empty result. In the "Repeat WHILE" mode, the action returns the 1st non-empty result.

Frequently (but not always), the called module/project may need to have a conditional branching at some point. The condition would serve as the loop exit condition. One of the branches should return an empty (or not empty) dataset, depending on the loop type.

If the called module contains the "Input" action, on the 1st iteration it's populated with the input dataset of the "Repeat" action that calls it. In consequent iterations, it's populated with the result dataset from the previous iteration. In other words, the result of the previous iteration is automatically passed as input to the next iteration.

The "Repeat" action can help in numerous cases:

• Obtaining data from REST APIs with paging (example)
• Polling a web API until an external task is complete
• Dealing with parent-child data structures (example)
• Repeatedly apply the same calculation to a dataset until it no longer changes
• Wait until a file appears in a folder

Nested iterations (not available in the free edition)

An iterated module can in turn iterate another module, effectively forming a nested iteration. In EasyMorph, nested iterations can be of any depth. However, recursion or cyclical dependencies are not allowed.

Note that in the "Iterate" action, the iterated module itself can be specified using a parameter. Therefore, using nested iterations it is possible to execute a list of EasyMorph modules or projects, collecting their results into one table. This workflow is helpful when you have many modules/projects that have customized action logic (e.g. adjusted for each customer), but return standardized results. For instance, you need to collect balance sheets from multiple counterparties, but each counterparty sends it in a different format. Therefore, you can use a separate EasyMorph module (or project) for each counterparty to transform its balance sheet into some uniform representation of the balance sheet, and then append all uniform balance sheets into one dataset.