Version 5.9.7 is the last planned release for version 5 of EasyMorph. It brings lots of new and improved features to both EasyMorph Desktop and EasyMorph Server. Let's take a look at some of the best bits.
Python Integration and the Call Python action
It has always been possible to run Python scripts from EasyMorph workflows, by execution them using the "Run program" action. However this was less than ideal and more of a "fire and wait" approach. In version 5.9.7 we've added several new features allowing EasyMorph and Python scripts to integrated tightly together. The first is the new "Call Python" action.
On it's own, the Call Python action might not seem like much of an improvement over the Run program action. However, we've also introduced the "easymorph" library for Python. This library opens up far more detailed and direct integration of Python into EasyMorph workflows. With this library you can:
- Access datasets passed from EasyMorph to the Python script.
- Pass back datasets from the Python script to the EasyMorph workflow it was called from.
- Access the parameters of the EasyMorph workflow which called the script.
- Reported back the status or errors to the EasyMorph workflow.
- Open, edit and create EasyMorph datasets as well as convert them to and from Pandas DataFrames.
This integration between Python and EasyMorph opens up new possibilities for combining data science and machine learning into your existing and future data preparation and ETL pipelines. As well as bringing together data engineering and data science teams whilst keeping the tools and technologies they know and use daily. For more information on the Python integration and what it can do, take a look at our recent post announcing it and the EasyMorph Community.
NOTE: This integration of Python into EasyMorph is brand new and considered an experimental feature. As such, we have more improvements and performance enhancements planned for future releases.
New action - Skip if unchanged
This release brings a second new action, the "Skip if unchanged" action. Unlike the other skip actions which rely on fixed conditions, this action allows you to specify one or more columns to monitor for changes.
To make it possible to check if the data has changed, EasyMorph automatically caches a copy of the required data into a .dset file in the same folder as the EasyMorph project is located. Each time the workflow runs, EasyMorph checks whether or not the data has changed compared to this cached version and if it hasn't, just like the other skip actions, the rest of the actions in the table are skipped. If the data has changed, the cache is updated and the rest of the actions are not skipped.
The action has two main "change detection modes". The first checks for and returns rows which have been added or modified since the previous run. This mode is useful for identifying and processing only the records that have changed, making your ETL pipelines more efficient. Combined with the existing "Select matching database rows action, incremental loading of data from databases is easier than ever before.
The other change detection mode checks for and returns only rows which have been removed since the previous run. This mode is most useful when performing incremental loading of data from a database or source system, making it simple to identify the records that have been deleted so that we can remove them from the already loaded data.
TIP: Whilst we haven't added a matching "Halt/warn if unchanged" action, this can easily be achieved. By default, the Skip if unchanged action outputs an empty dataset if the data hasn't changed. Placing the "Halt/warn on condition" action immediately after it and setting the condition to "Halt if the table is empty" results in the entire workflow run being halted if nothing has changed.
Email command action improvements
In version 5.9.7 we've added a new option to the "Email command" action allowing you to "List folders" within a connected email account.
Now you can create a workflow which automatically loops through all available folders in an email account, fetching emails from them, even if folders are added or removed.
Annotations for Groups
Whilst it was possible to add annotations to just about every element of an EasyMorph project (actions, tables, modules, etc), somehow Groups (also known as tabs) were left out. I know it's far from the most impressive new feature, but we've added them in this release.
Now when you hover over the Group you will see the annotation.
Issues pagination on Explorer Boards
EasyMorph Server has also received some updates, starting with Issues sections on Explorer Boards. Previously, if a large number of issues were added to a Board it would result in the Board simply becoming longer and longer. In version 5.9.7 we've added pagination so that if you add more than 50 issues to a Board they will flow onto different pages instead.
Server Catalog Improvements
A big theme for recent releases has been to close the gap between what is possible to add to the Explorer Catalog in EasyMorph Desktop but which couldn't be added directly from the EasyMorph Server web UI. This release continues the trend with several significant improvements.
Dataset commands in the Server UI
You can now add and edit Commands for dataset Assets in the Catalog directly. Now Row, Column and General commands can all be added by clicking on the 3 dots in the top right corner of any computed or pre-computed dataset in the Catalog and selection "Add/edit commands".
NOTE: "Static datasets" were renamed "Pre-computed datasets" in this release to better reflect that they are not necessarily static and can still change if the workflow which created the dataset is run again.
For those not aware of Commands in the Catalog, I like to think of them as adding hyperlinks to a dataset asset which link to any other Catalog asset. Because there are many types of Catalog assets, the possibilities for Commands are almost endless. Some of my favorite use cases for them are:
- Allow users to navigate from one dataset to another, such as clicking on a client's name and retrieving all orders for that client. Then clicking on an order ID and being taken directly to the order in the CRM system to they can see all of the details.
- Allow users to change the status of a record by clicking on the status, selecting the desired status and running a workflow passing the new status as a parameter to update the record in the source system.
- Providing a link so users can click to create an Issue in the Issues section of a Board to alert someone to a problem.
- Open a Tableau Workbook or PowerBI dashboard containing the same data.
- At the click of a button, allow the finance team to release the month end data to the rest of the business once they are happy it is reconciled.
In this release we've also added support for Column Commands in both the Analysis View and in Table sections on EasyMorph Server Boards. General Commands are also now supported in the Analysis View. The first General Command will appear as a button, with any additional General Commands available in a dropdown menu accessible by clicking on the "More" button.
Related Assets in the Server UI
And to top it off, you can also now add and edit "Related Assets" in the settings for all Catalog assets directly in the Server web UI. When adding or editing an Asset, click on the "Related Assets" tab to configure related Assets.
More permissions for Explorer
Last but not least, version 5.9.7 brings additional permissions for EasyMorph Explorer Boards and the Catalog allowing more granular control over who can:
- See, retrieve, create and edit computed and pre-computed assets.
- Create, edit and delete Boards.
- Manually create, modify, forward and snooze issues.
- Archive and restore issues.
- Download datasets from the Analysis View.
What's coming next?
DISCLAIMER: The below is for informational purposes only. It does not guarantee that any mentioned features will be implemented in the software or included in a specific version. Plans and priorities may change at any time.
As mentioned at the start of this post, Version 5.9.7 is the last planned release for EasyMorph version 5. Version 6 is expected some time in the last quarter of this year.
There are a few main themes for version 6:
- Eliminating the gap - Continuing to close the gap so that everything that can be configured in the Catalog in EasyMorph Desktop can also be done directly in the EasyMorph Server web interface.
- Modernize - EasyMorph Server is getting a brand new and updated user interface bringing a more modern look and feel and making it easier to navigate and work within, whatever a users technical ability or experience with EasyMorph. We might even allow you to change the look and feel to match your organizations branding.
- Foundations - We're upgrading the underlying technology behind EasyMorph which will not only bring some performance improvements but will also lay the foundations for some of the exciting new features we have planned for next year.
- Marching forward - And of course, we'll continue to bring new and exciting features and improvements to both EasyMorph Server and EasyMorph Desktop so that it continues to be the data engineering and automation platform you all love.
