Iterations (loops)

Loops in EasyMorph are arranged with the help of iterations. Using iterations you can perform various repetitive workflows — e.g. load multiple uniform files, process data for a range of dates, or run the same calculation but with different parameters several times in order to evaluate various scenarios (a.k.a. "what-if" analysis).

In a nutshell, here is how iterations work:
  1. 1) create a list
  2. 2) perform the same action with each item of the list

To explain it better, here is an analogy:

Let's assume that Alice needs to deliver a few letters to their recipients, and she has a list of their addresses. She asks Bob to help her with this task. Bob can deliver a letter — one letter to one recepient (that's the action with a list item). Alice gives Bob a letter, tells him where to deliver it, and Bob does it. It repeats for all addresses on Alice's list, from first to last address.

Here is the same story, told in EasyMorph terms: Project "Alice" has a table with a list of addresses. Project "Bob" has a parameter named Address, and delivers one letter to the address specified by the parameter. For each actual address in the list project "Alice" assigns the address to "Bob's" parameter Address, and then runs project "Bob".

That's how iterations work in EasyMorph — one project runs (calls) another project once for every row of a table. Parameters of the iterated project are assigned using field values of the table from corresponding table row. Notice that the action (from the definition above) is an EasyMorph project which means that in a simple case the action can be just one transformation (i.e. the iterated project would have only one transformation, e.g. loading a file), or it can be many transformations that implement a rather complex calculation logic.

The diagram below shows how Project A iterates Project B across a list of three files. It runs Project B three times — once per each file name in column [File Name]. Project B has a parameter named Param1. This parameter is assigned by Project A when it runs Project B.

Most frequently, iterations are run for lists of files, or lists of dates. Such lists can be generated using transformation "Calendar" (generates a sequence of dates) or transformation "List of files" (generates a list of files in a folder). Alternatively, they can be imported from a database or a text file.

Hint: To create a FOR..NEXT type of loop, generate a sequence of numbers 1..N using "Sequence" transformation. Then iterate across the sequence.

"Iterate" transformation

"Iterate" transformation performs iterations for a table. It runs another project once for each line of the table. Parameters of the iterated project are assigned using field values of the table. If a project parameter is not assigned, it keeps last saved value (the default value).

"Iterate" transformation can run in two modes:
  • Iterate — do not return anything from the iterated project, just run it
  • Iterate and append — return a table (the result table) from each iteration, and append all result tables into one table, when all iterations are complete; this table becomes the transformation's output

When you need to load multiple uniform files, create a helper project that loads one file which name is defined by a project parameter. In the simplest case this project will consist of only one import transformation and one parameter. Then in the main project iterate through a list of files (e.g. created with the help of "List of files" transformation), running the helper project using "Iterate" transformation in the "Iterate and append" mode. The result of "Iterate" transformation will be appended data from all files (see also Example 1 below).

Hint: To quickly obtain a list of files in a folder simply drag the folder into EasyMorph.

"Iterate table" transformation (ver.3.1 or above)

"Iterate table" transformation is basically the same as "Iterate" transformation with the only addition that besides assigning project parameters of the iterated project it also passes another table into it. The iterated project should use "Input" transformation in order to obtain the table that is passed to it by the iterating project. This transformation is helpful when multiple calculations with different parameters have to be performed on the same dataset (e.g. financial stress testing).

Hint: To run another EasyMorph project once, use "Iterate" transformation for a table with only 1 row, or use "Call" transformation.

Examples

We've preperared a pack of 5 iteration examples. Each example has an illustrated description and annotations in project tables and transformations.

  • Example 1 – load multiple files
  • Example 2 – split one file into many files
  • Example 3 – batch processing of several files
  • Example 4 – iterate and append data into one table
  • Example 5 – iterate another table (requires ver.3.1 or above)

Download iteration examples

"Iterate program" transformation

"Iterate program" transformation is similar to "Iterate" transformation, but instead of running another EasyMorph project it runs an external application once for each line of a table. The command line in this case is specified as expression, therefore it can be calculated for each line of the table using fields, functions and parameters. If the external application is not defined, the transformation executes the command line as if it is a Windows shell command. It can be used, for instance, for mass copying/deleting/moving files.

Nested iterations (Professional edition only)

We can extend our analogy with Alice and Bob with one more level of iteration:

Let's assume that Alice received the list of addresses from Pete, and Pete has several lists of addresses. Pete gives Alice the lists one by one, and Alice arranges delivery with Bob's help. This scenario describes nested iterations — when an iteration is performed inside a step of another iteration. For instance, outer iteration runs across a list of dates, and inner iteration runs across a list of countries, processing multiple countries for each date from the list. In EasyMorph nested iterations can be of any depth.

Note that in "Iterate" transformation the iterated project itself can be specified using a parameter. Therefore, using nested iterations it is possible to execute a list of EasyMorph projects, collecting their results into one table. This workflow is helpful when you have many projects that have customized transformation logic (e.g. adjusted for each customer), but return standardized results. For instance, you need to collect balance sheets from multiple counterparties, but each counterparty sends it in a different format. Therefore you use a separate EasyMorph project for each counterparty to transform its balance sheet to a some uniform representation, and then append all uniform balance sheets into one dataset (table).


Read next: Reusable projects