Connectors - general functionalities 

In this article are described general functionalities for managing native connectors in solution. All Native Connectors are managed from the same connector management UI. 

Notice that there are separate descriptions for each connector, where connector specific functionalities and configuration instructions are described in detailed level. 

To be able to access connector management, user needs to have admin level permissions to customers Platform configuration. When accesses are granted correctly, connector tab will be visible and user can manage connectors. 

Left menu

Connector management is divided into four tabs: 

• Overview - for creating and updating Native Connectors. The Admin User can see their status, type and how many scheduled or event tasks are associated with them.
• Authentication - for creating and updating authentication tasks. Provisioning task for authentication is needed for Secure Access to be able to define which Customers end-users can access to Matrix42 Core, Pro and IGA login page. 
• Logs - for downloading Native Connector and Secure Access logs from UI.
• Settings - general settings for Native Connectors and Secure Access, including environment type for logging and monitoring.

Connectors overview tab

From overview page, user can easily and quickly see status for all connectors. 

Top bar:

• Status for Native Connectors (EPE)
  • Green text indicates that Native Connectors is online. All the needed services are up and running.
  • Red text indicates that there is a problem with Native Connectors, all the services are not running.
• Status for Secure Access (ESA)
  • Green text indicates that Secure Access is online. All the needed services are up and running.
  • Red text indicates that there is a problem with Secure Access, all the services are not running.
• Native Connectors version number is displayed in top right corner
   

Top bar for list view:

• New connector - opens new window for adding and configuring new connector 
• Remove connector(s) - workflow references are calculated and pop-up window appears to confirm removal (notice that calculating references can take several seconds)
• Export - Admin can export one or more connectors (and tasks) from the environment. Usually used for exporting connectors and connectors (and tasks) from test to prod. Native connectors secret information is password protected.
• Import - Admin can import one or more connectors (and tasks) to the environment. Usually used for importing connectors (and tasks) from test to prod. 
  • Admin cannot import from old EPE UI (Older than 2024.1) to the new one. Source and target environments must have same version.
  • Import will fail if the configuration (templates, attributes) is not the same - for example when some attribute is missing 
  • If you are importing something with the same connector details, it will be merged under existing connector
• Refresh - Admin can refresh connectors view by clicking the button.

List view for overview, 

• Select connector(s) - Select one connector by clicking check box in front of the connector row or clicking check box in the header row will select all connectors
• Id - Automatically generated unique ID of the connector. Cannot be edited or changed.
• Status - indicates scheduled task status
  • Green check mark - Task is executed without any errors
  • Red cross -Task is executed, but there have been error
  • Grey clock - Task is not executed yet, waiting for scheduling
  • Orange - one of the tasks has a problem
  • No value - Scheduled based-task is missing
• Name - Connector name added to connector settings. Unique name of the connector holding configuration for one data source
• Type - indicates target / source system
• Scheduled - informs how many scheduled tasks are configured
• Event - informs how many event tasks are configured
• Manage
  • Pen icon - opens connector settings (double-clicking the connector row, also opens settings)
  • Paper icon - copies the connector
  • Stop - workflow references are calculated and pop-up window appears to confirm removal
• Search - User can search connector by entering the search term in the corresponding field . Id, Status, Name, Type, Scheduled and Event fields can be searched.

Scheduled task information (click arrow front of the connector)

When clicking arrow at the beginning of the connector row, all related scheduled and event tasks are shown

Top bar for scheduled tasks

• New task - opens configuration page for new task
• Remove task(s) - removes the selected task(s) from the system and they cannot be recovered anymore.
• Refresh - refresh scheduled-based tasks view 
• Search - user can search task by entering the search term in the corresponding field for Id, name, enabled, extract/load status.

List view for scheduled tasks

• Select task(s) - Select task to be deleted from the list view by ticking.
• Id - Unique ID of the task. Generated automatically and cannot be changed.
• Name - Task name added to task settings, unique name of the task.
• Enabled - Displays is the task scheduled or not
  • Green check mark - Task is scheduled
  • Red cross - Task is not scheduled
• Extract status - Displays status of data extraction from the target directory/system
  • Green check mark - data is extracted successfully
  • Red cross - data is extracted with error(s) or extraction is failed 
  • Clock - Task is waiting execution
• Load status - Displays status of data export from json-file to customers solution 
  • Green check mark - data is imported successfully to customers solution
  • Red cross - data is imported with error(s) or import is failed 
• Manage
  • Pen icon - opens task settings in own window (double-clicking the task row also opens task settings)
  • Paper icon - copies the task
  • Clock icon - opens task history view
  • Stop - remove task, pop-up window opens to confirm the removal

Scheduled task history view 

By clicking the clock icon in the scheduled task row, history for scheduling will be shown.

Top bar for view history

• Refresh - refreshes scheduled task status. This doesn't affect task, this only updates UI to show latest information of task run.

List view for scheduled task history 

• Color of the row is indicating status
  • Green - task executed successfully
  • Red - error happened during execution
• Execution ID - unique ID for the scheduled task row
• Extract planned time - when next extract from the directory/application is scheduled to happen
• Extract complete time - when extract was completed
• Extract status - status for fetching data from the directory/application
• Load start time - when next load to customers solution is scheduled to happen
• Load complete time - when load was completed
• Load status - status for loading information to customers solution

List view for scheduled task status

• Actual start time - timestamp for actual start
• Users file - JSON file containing user information read from the directory/application
• Group file - JSON file containing group information read from the directory/application
• Generic file - JSON file containing generic information read from the directory/application
• Extract info - detailed information about reading information from the directory/application
• Load info - detailed information about loading the information to customers Matrix42 Core, Pro and IGA solution

Edit window for scheduled task

Configuration for scheduled task can be opened by clicking pen icon or double-clicking the task row. 

Left menu and attributes varies according to selected options and therefore more detailed instructions for editing tasks can be found from the connector description, but there are common functionalities for all scheduled tasks which are described below. 

Saving the task

In case mandatory information is missing from the task, hoovering mouse on top of the save button will show which attributes are still empty. 

Top bar for edit scheduled task

• Run task manually - admin can run task manually out side of the defined scheduling
• Stop task - admin can stop scheduled-based task which is currently running. Task will be stopped and status is changed to be stopped. It waits in this state until the next timing occurs. 
• Clear data cache -  Data cache for the next provisioning of Users and Groups will be cleared. It means that next run is run as first time run. 
  • By default, we clear the cache everyday at 00:00 UTC
  • If you want to clear the cache at different time, then it has to specify some different value in host file 'custom.properties'. 
  • EPE Cache is also cleared when EPE is restarted, whole environment is restarted, EPE mappings have been changed
     

Event task information

When clicking arrow at the beginning of the connector row, all related scheduled and event tasks are shown

Top bar for event tasks

• New task - opens configuration page for new event task
• Remove task(s) - removes selected task(s), pop-up window appears to confirm the removal
• Refresh - refreshes event tasks view 
• Show workflow references - calculates task related workflow relations and statuses. This is very useful if you don't know from which workflows event-based tasks are used.

List view for event tasks

• Select task(s) - Select task to be deleted from the list view by ticking.
• Id -  Unique ID of the task. Generated automatically and cannot be changed.
• Name - Task name added to task settings, unique name of the task.
• Workflow relations
  • Question mark shows pop-up window with detailed information about the reference
• Workflow status
  • Not used - No relations to workflow
  • In use -  Workflow(s) attached to task, task cannot be removed 
• Manage
  • Pen icon - opens task settings in own window
  • Paper icon - copies the task
  • Stop icon - removes the task, pop-up window appears to confirm the removal

Edit window for event task

Configuration for event task can be opened by clicking pen icon or double-clicking the task row. 

Left menu and attributes varies according to selected options and therefore more detailed instructions for editing tasks can be found from the connector description, but there are common functionalities for all event tasks which are described below. 

Edit event task window

• Task usage, editable? - this appears when editing existing task and changing the task usage type will break workflows
• Mappings type, editable? - this appears when editing existing task and changing the mappings type will break workflows

Saving the task

In case mandatory information is missing from the task, hovering mouse on top of the save button will show which attributes are still empty. 

Authentication tab

Authentication for Matrix42 Core, Pro and IGA solutions are configured from authentication tab, notice that only some of the connectors (directory connectors) are supporting authentication, so its not possible to create authentication tasks to all available connectors.

Top bar for authentication

• New connector - opens new window for configuring new connector (notice that not all connectors are supporting authentication)
• Remove connector(s) - removes selected task(s), pop-up window appears to confirm the removal
• Export - user can export one or more tasks from the environment. Usually used for exporting tasks from test to prod. EPE connectors are password protected.
  • Note that Realm for authentication tasks is not exported, you need to set that manually after importing
• Import - user can import one or more tasks to the environment. Usually used for importing task from test to prod. 
• Refresh - refreshes authentication tasks view 

List view for authentication overview

• Select connector(s) - Select one connector by clicking check box in front of the connector row or clicking check box in the header row will select all connectors
• Id - Automatically generated Unique ID of the connector. Cannot be edited or changed.
• Name - Connector name added to connector settings. Unique name of the connector holding configuration for one data source
• Type - indicates target / source system
• Count -  informs how many authentication tasks are configured
• Manage
  • Pen icon - opens authentication task setting in own window
  • Paper icon - copies the task
  • Stop icon - removes selected task

Authentication task information

When clicking arrow at the beginning of the connector row, all related scheduled and event tasks are shown

Top bar for authentication overview

• Create new task - opens configuration page for new authentication task
• Remove task(s) - removes selected task(s), pop-up window appears to confirm the removal
• Refresh -  refreshes authentication tasks view 

List view for authentication overview, 

• Select task(s) - Select task to be deleted from the list view by ticking.
• Id - Unique ID of the task. Generated automatically and cannot be changed.
• Name - Task name added to task settings, unique name of the task.
• Manage
  • Pen icon- opens task settings in own window (double-clicking the task row, also opens settings window)
  • Paper icon - copies the task
  • Stop icon - removes the task, pop-up window appears to confirm the removal

Logs tab

Logs tab is for downloading Native Connector and Secure Access logs from UI for detailed troubleshooting.

• epe-master logs - contains warning, debug and error level messages about Native Connectors and info how long task actions has been taken. 
• epe-worker-ad logs - contains extract data status of Active Directory connector (what Native Connector is loading to to customers Matrix42 Core, Pro and IGA solution). If selection is empty, it means that directory is not in use in this environment. 
• epe-worker-azure logs - contains extract data status of Entra ID (what Native Connector is loading to to customers Matrix42 Core, Pro and IGA solution). If selection is empty, it means that directory is not in use in this environment. 
• epe-worker-ldap logs - contains extract data status of LDAP (what Native Connector is loading to to customers Matrix42 Core, Pro and IGA solution). If selection is empty, it means that directory is not in use in this environment. 
• epe-launcher logs - contains information about EPE launches
• datapump-itsm logs - Contains information about data export to customers Matrix42 Core, Pro and IGA solution.
• esa logs - Contains information about Secure Access authentication. 

Settings tab

Settings tabs is used for monitoring environments with connectors. 

• Environment type - is mandatory to be in selected and information is used for example defining alert critically. 
  • Test - select this when your environment is used as testing environment
  • Prod - select this when your environment is used as production environment
  • Demo - select this when your environment is used as demo or training environment
  • Dev - select this when your environment is used as development environment

What we are monitoring?

• Failures in scheduled based provisioning (extracting data, exporting data to ESM, outdated certificates, incorrect passwords, incorrect search base/filter, incorrect mappings, etc.)
• Failures in event based provisioning (fail to write to AD/Azure, etc.)
• Event-based provisioning - which connectors are used for writing data towards applications/directories. 
• ESA more than ten failed login attempts to one user in past 3 days
• Environment type - is mandatory to be in selected and information is used for example defining alert critically.

Data migrations

Do not click “Migrate attribute mappings” or “Migrate workflows”, if not requested to do so by Matrix42.

General info

Generic Python Script connector comes with couple out-of-the box Python scripts. You should not modify those. Instead, if you need to make modifications to those, copy-paste those to new name and then do your modifications to newly named python script file. And as last step, modify your connector to use that script.

Work order:

1. Create your custom scheduled or task-based script
2. Take it into use to connector from Connectors UI

Scheduled scripts

How to take your custom python script into use for scheduled tasks, guidance for 1 host and 1 worker environment:

Note! Do not change "default" .py files which came with environment installation.

Copy your customly named (in this example mypythonscript.py) python script to these folders from HOST (remember to use correct tenant_name in path):

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_scheduled

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-worker-1/files/custom-provisioning-scripts

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts

Change your custom python script file permission to 755 on all those above locations where you copied your script(in this example mypythonscript.py) (remember to use correct tenant_name in path):

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_scheduled/mypythonscript.py

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-worker-1/files/custom-provisioning-scripts/mypythonscript.py

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/mypythonscript.py

If you have more hosts, repeat on every host.

If you have more workers, repeat to every worker(check that worker folder name is correct).

Event-based scripts

How to take your custom python script into use for event-based tasks, guidance for 1 host environment:

Note! Do not change "default" .py files which came with environment installation.

Copy your customly named (in this example mypythonscript.py) python script to these folders from HOST (remember to use correct tenant_name in path):

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_event

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts

Change your custom python script file permission to 755 on all those above locations where you copied your script(in this example mypythonscript.py) (remember to use correct tenant_name in path):

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_event/mypythonscript.py

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/mypythonscript.py

If you have more hosts, repeat on every host.

#!/usr/bin/python3

import json
import sys
import logging

def main():
    # Configure logging, change log file name
    logging.basicConfig(filename='python_event_task_abc.log', level=logging.INFO, format='%(asctime)s - %(message)s')
    
    # Read inputs from stdin
    lines = sys.stdin.read().splitlines()

    if len(lines) != 2:
        print("Error: Expected exactly two lines of input.")
        raise Exception("Expected exactly two lines of input.")

    # Parse the first line, this contains connector variables
    try:
        connector_parameters = json.loads(lines[0])
        host = connector_parameters.get('host', None)
        password = connector_parameters.get('password', None)
        logging.info(f"Connector parameters, Host={host}")
        # read custom connector parameters here
    except json.JSONDecodeError:
        raise Exception("Connector parameters is not a valid JSON string.") 

    # Parse the second line, this contains mappings data from template datacard
    try:
        task_mappings = json.loads(lines[1])
        logging.info(f"Task mappings JSON: {task_mappings}")
        #read task mappings data here

        # Implement your logic here

        
    except json.JSONDecodeError as e:
        raise Exception("Task mappings is not a valid JSON string.") from e
        # Exceptions raised are controlling workflow orchestration node flow, on exception it goes to Exception path, otherwise it goes to Completed path. Exception raised can be seen on Provisioning exception attribute value.
    except Exception as e:
        raise Exception("An unexpected error occurred") from e

if __name__ == "__main__":
    main()

Take customly named script into use to connector

How to take your customly named Python script into use, by selecting it from Connectors UI. Select your script from connectors Provisioning script -dropdown: