Connectors - general functionalities 

In this article are described general functionalities for managing native connectors in solution. All Native Connectors are managed from the same connector management UI. 

Notice that there are separate descriptions for each connector, where connector specific functionalities and configuration instructions are described in detailed level. 

To be able to access connector management, user needs to have admin level permissions to customers Platform configuration. When accesses are granted correctly, Connector tab will be visible and user can manage connectors. 

Left menu

Connector management is divided into four tabs: 

• Overview - for creating and updating Native Connectors. The Admin User can see their status, type and how many scheduled or event tasks are associated with them.
• Authentication - for creating and updating authentication tasks. Provisioning task for authentication is needed for Secure Access to be able to define which Customers end-users can access to Matrix42 Core, Pro and IGA login page. 
• Logs - for downloading Native Connector and Secure Access logs from UI.
• Settings - general settings for Native Connectors and Secure Access, including environment type for logging and monitoring.

In release 26.2 left menu was moved to be on top of page as top menu. It also now contains new Certificates tab, for handling Native Connectors and Secure Access LDAP and HTTPS traffic certificates.

Connectors overview tab

From overview page, user can easily and quickly see status for all connectors. 

Top bar:

• Connectors
  • Green indicates that Native Connectors is online. All the needed services are up and running.
  • Red indicates that there is a problem with Native Connectors, all the services are not running.
• Authentication (Secure Access)
  • Green indicates that Secure Access is online. All the needed services are up and running.
  • Red indicates that there is a problem with Secure Access, all the services are not running.
• Native Connectors version number is displayed in top right corner
   

Top bar for list view:

• New connector - opens new window for adding and configuring new connector 
• Remove connector(s) - workflow references are calculated and pop-up window appears to confirm removal (notice that calculating references can take several seconds)
• Export - Admin can export one or more connectors (and tasks) from the environment. Usually used for exporting connectors and connectors (and tasks) from test to prod. Native connectors secret information is password protected.
• Import - Admin can import one or more connectors (and tasks) to the environment. Usually used for importing connectors (and tasks) from test to prod. 
  • Admin cannot import from old EPE UI (Older than 2024.1) to the new one. Source and target environments must have same version.
  • Import will fail if the configuration (templates, attributes) is not the same - for example when some attribute is missing 
  • If a connector with the same details already exists in the environment, the import process detects this and informs the admin that the connector already exists. The existing connector is not modified or renamed.
• Refresh - Admin can refresh connectors list view by clicking the Refesh button.

List view for overview, 

• Select connector(s) - Select one connector by clicking check box in front of the connector row or clicking check box in the header row will select all connectors
• Id - Automatically generated unique ID of the connector. Cannot be edited or changed.
• Status - indicates scheduled task status

• Green double check mark - Task is executed without any errors
• Red triangle -Task is executed, but there have been error
• Grey line - Tasks are not executed yet
• Orange - one or more of the tasks has a problem
• No value - Scheduled based-task is missing
• Name - Connector name added to connector settings. Unique name of the connector holding configuration for one data source
• Type - indicates target / source system
• Scheduled - informs how many scheduled tasks are configured
• Event - informs how many event tasks are configured
• Manage
  • Pen icon - opens connector settings (double-clicking the connector row, also opens settings)
  • Paper icon - copies the connector
  • Stop - workflow references are calculated and pop-up window appears to confirm removal
• Search - User can search connector by entering the search term in the corresponding field . Id, Status, Name, Type, Scheduled and Event fields can be searched.

Scheduled task information (click arrow front of the connector)

When clicking arrow at the beginning of the connector row, all related scheduled and event tasks are shown

Top bar for scheduled tasks

• New task - opens configuration page for new task
• Remove task(s) - removes the selected task(s) from the system and they cannot be recovered anymore.
• Refresh - refresh scheduled-based tasks view 
• Search - user can search task by entering the search term in the corresponding field for Id, name, enabled, extract/load status.

List view for scheduled tasks

• Select task(s) - Select task to be deleted from the list view by ticking.
• Id - Unique ID of the task. Generated automatically and cannot be changed.
• Name - Task name added to task settings, unique name of the task.
• Scheduling - Displays is the task scheduled or not
  •   Green calendar mark - Task is scheduled
  • Grey line - Task is not scheduled
• Extract status - Displays status of data extraction from the target directory/system
  •   Green double check mark - data is extracted successfully
  • Red triangle - data is extracted with error(s) or extraction is failed 
  • Grey line - Extract has not been executed yet
  • Black gears - Extraction is ongoing
• Load status - Displays status of data export from json-file to customers solution 
  •   Green double check mark - data is imported successfully to customers solution
  • Red triangle - data is imported with error(s) or import is failed 
  • Grey line - Data load has not been executed yet
  • Black gears - Load is ongoing
• Manage

• Pen icon - opens task settings in own window (double-clicking the task row also opens task settings)
• Paper icon - copies the task
• Move icon - move task under some different connector, pop-up window opens to select target connector
• Run icon - run task manually once, pop-up window opens to confirm the run
• Clock icon - opens task history view
• Minus icon - remove task, pop-up window opens to confirm the removal
   

Scheduled task history view 

By clicking the clock icon in the scheduled task row, history for scheduling will be shown.

It shows maximum 500 previous task runs.

Top bar for view history

• Refresh - refreshes scheduled task status. This doesn't affect task, this only updates UI to show latest information of task run.

List view for scheduled task history 

• Color of the row is indicating status
  •  Green double check mark - execution successful
  • Red triangle - execution failed (hover red icon to see more detailed error)
• Execution ID - unique ID for the scheduled task row
• Extract status - status for fetching data from the directory/application
• Load status - status for loading information to customers solution
• Scheduled start time - when next extract from the directory/application is scheduled to happen
• Extract actual start time - when extract was started
• Extract actual end time - when extract ended
• Load actual start time - when load actually started
• Load actual end time - when load actually ended

Task run details

• Download Group file - JSON file containing group information read from the directory/application
• Download Users file - JSON file containing user information read from the directory/application
• Download Generic file - JSON file containing generic information read from the directory/application
• Download All Failed Data Cards - JSON file containing failed objects
• Load overview - detailed information about loading the information to customers Matrix42 Pro and IGA solution
• Log data - part of logs. Full logs can be downloaded from Logs tab

Edit window for scheduled task

Configuration for scheduled task can be opened by clicking pen icon or double-clicking the task row. 

Left menu and attributes varies according to selected options and therefore more detailed instructions for editing tasks can be found from the connector description, but there are common functionalities for all scheduled tasks which are described below. 

Saving the task

In case mandatory information is missing from the task, hoovering mouse on top of the save button will show which attributes are still empty. 

Top bar for edit scheduled task

• Run task manually - admin can run task manually out side of the defined scheduling
• Stop task - admin can stop scheduled-based task which is currently running. Task will be stopped and status is changed to be stopped. It waits in this state until the next timing occurs. 
• Clear data cache -  Data cache for the next provisioning of Users and Groups will be cleared. It means that next run is run as first time run. 
  • By default, we clear the cache everyday at 00:00 UTC
  • If you want to clear the cache at different time, then it has to specify some different value in host file 'custom.properties'. 
  • EPE Cache is also cleared when EPE is restarted, whole environment is restarted, EPE mappings have been changed
     

Event task information

When clicking arrow at the beginning of the connector row, all related scheduled and event tasks are shown

Top bar for event tasks

• New task - opens configuration page for new event task
• Remove task(s) - removes selected task(s), pop-up window appears to confirm the removal
• Refresh - refreshes event tasks view 
• Show workflow references - calculates task related workflow relations and statuses. This is very useful if you don't know from which workflows event-based tasks are used.

List view for event tasks

• Select task(s) - Select task to be deleted from the list view by ticking.
• Id -  Unique ID of the task. Generated automatically and cannot be changed.
• Name - Task name added to task settings, unique name of the task.
• Workflow relations
  • Question mark shows pop-up window with detailed information about the reference
• Workflow status
  • Not used - No relations to workflow
  • In use -  Workflow(s) attached to task, task cannot be removed 
• Manage
  • Pen icon - opens task settings
  • Paper icon - copies the task
  • Move icon - moves task under different connector, pop-up window opens to select target connector
  • Stop icon - removes the task, pop-up window appears to confirm the removal

Edit window for event task

Configuration for event task can be opened by clicking pen icon or double-clicking the task row. 

Left menu and attributes varies according to selected options and therefore more detailed instructions for editing tasks can be found from the connector description, but there are common functionalities for all event tasks which are described below. 

• Task name - Descriptive name for task so you recognize it
• Task usage - Allow editing checkbox. When unchecked, the current task usage value is displayed in read-only form without edit indicators, so the configuration state is easy to read. Check the box to make the field editable, note that changing the task usage type on an existing task will break workflows
• Mappings type - Allow editing checkbox. When unchecked, the current mappings type value is displayed in read-only form without edit indicators. Check the box to make the field editable, note that changing the mappings type on an existing task will break workflows

Saving the task

In case mandatory information is missing from the task, hovering mouse on top of the save button will show which attributes are still empty. 

Authentication tab

Authentication for Matrix42 Core, Pro and IGA solutions are configured from authentication tab, notice that only some of the connectors (directory connectors) are supporting authentication, so its not possible to create authentication tasks to all available connectors.

Top bar for authentication

• New connector - opens new window for configuring new connector (notice that not all connectors are supporting authentication)
• Remove connector(s) - removes selected task(s), pop-up window appears to confirm the removal
• Export - user can export one or more tasks from the environment. Usually used for exporting tasks from test to prod. EPE connectors are password protected.
  • Note that Realm for authentication tasks is not exported, you need to set that manually after importing
• Import - user can import one or more tasks to the environment. Usually used for importing task from test to prod. 
• Refresh - refreshes authentication tasks view 

List view for authentication overview

• Select connector(s) - Select one connector by clicking check box in front of the connector row or clicking check box in the header row will select all connectors
• Id - Automatically generated Unique ID of the connector. Cannot be edited or changed.
• Name - Connector name added to connector settings. Unique name of the connector holding configuration for one data source
• Type - indicates target / source system
• Count -  informs how many authentication tasks are configured
• Manage
  • Pen icon - opens authentication task setting in own window
  • Paper icon - copies the task
  • Stop icon - removes selected task

Authentication task information

When clicking arrow at the beginning of the connector row, all related scheduled and event tasks are shown

Top bar for authentication overview

• Create new task - opens configuration page for new authentication task
• Remove task(s) - removes selected task(s), pop-up window appears to confirm the removal
• Refresh -  refreshes authentication tasks view 

List view for authentication overview, 

• Select task(s) - Select task to be deleted from the list view by ticking.
• Id - Unique ID of the task. Generated automatically and cannot be changed.
• Name - Task name added to task settings, unique name of the task.
• Manage
  • Pen icon- opens task settings in own window (double-clicking the task row, also opens settings window)
  • Paper icon - copies the task
  • Stop icon - removes the task, pop-up window appears to confirm the removal

Logs tab

Logs tab is for downloading Native Connector and Secure Access logs from UI for detailed troubleshooting.

• epe-master logs - contains warning, debug and error level messages about Native Connectors and info how long task actions has been taken. 
• epe-worker-ad logs - contains extract data status of Active Directory connector (what Native Connector is loading to to customers Matrix42 Pro and IGA solution). If selection is empty, it means that directory is not in use in this environment. 
• epe-worker-azure logs - contains extract data status of Entra ID (what Native Connector is loading to to customers Matrix42 Pro and IGA solution). If selection is empty, it means that directory is not in use in this environment. 
• epe-worker-ldap logs - contains extract data status of LDAP (what Native Connector is loading to to customers Matrix42 Pro and IGA solution). If selection is empty, it means that directory is not in use in this environment. 
• epe-launcher logs - contains information about EPE launches
• datapump-itsm logs - Contains information about data export to customers Matrix42 Pro and IGA solution.
• esa logs - Contains information about Secure Access authentication. 

Settings tab

Settings tabs is used for monitoring environments with connectors. 

• Environment type - is mandatory to be in selected and information is used for example defining alert critically. 
  • Test - select this when your environment is used as testing environment
  • Prod - select this when your environment is used as production environment
  • Demo - select this when your environment is used as demo or training environment
  • Dev - select this when your environment is used as development environment

What we are monitoring?

• Failures in scheduled based provisioning (extracting data, exporting data to ESM, outdated certificates, incorrect passwords, incorrect search base/filter, incorrect mappings, etc.)
• Failures in event based provisioning (fail to write to AD/Azure, etc.)
• Event-based provisioning - which connectors are used for writing data towards applications/directories. 
• ESA more than ten failed login attempts to one user in past 3 days
• Environment type - is mandatory to be in selected and information is used for example defining alert critically.

Show logs - when ON, logs can be seen from Logs tab on UI

Automate Secure Access Certificate reloading - when ON, Secure Access will automatically reboot at 03:00 UTC if there are new certificates that needs to be taken into use.

Data migrations - Do not click “Migrate attribute mappings” or “Migrate workflows”, if not requested to do so by Matrix42.

General info

Generic Python Script connector comes with couple out-of-the box Python scripts. You should not modify those. Instead, if you need to make modifications to those, copy-paste those to new name and then do your modifications to newly named python script file. And as last step, modify your connector to use that script.

Work order:

1. Create your custom scheduled or task-based script
2. Take it into use to connector from Connectors UI

Scheduled scripts

How to take your custom python script into use for scheduled tasks, guidance for 1 host and 1 worker environment:

Note! Do not change "default" .py files which came with environment installation.

Copy your customly named (in this example mypythonscript.py) python script to these folders from HOST (remember to use correct tenant_name in path):

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_scheduled

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-worker-1/files/custom-provisioning-scripts

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts

Change your custom python script file permission to 755 on all those above locations where you copied your script(in this example mypythonscript.py) (remember to use correct tenant_name in path):

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_scheduled/mypythonscript.py

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-worker-1/files/custom-provisioning-scripts/mypythonscript.py

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/mypythonscript.py

If you have more hosts, repeat on every host.

If you have more workers, repeat to every worker(check that worker folder name is correct).

Event-based scripts

How to take your custom python script into use for event-based tasks, guidance for 1 host environment:

Note! Do not change "default" .py files which came with environment installation.

Copy your customly named (in this example mypythonscript.py) python script to these folders from HOST (remember to use correct tenant_name in path):

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_event

cp mypythonscript.py /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts

Change your custom python script file permission to 755 on all those above locations where you copied your script(in this example mypythonscript.py) (remember to use correct tenant_name in path):

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/only_event/mypythonscript.py

chmod 755 /var/lib/efecteone/tenant_files/{tenant_name}/epe-master/files/custom-provisioning-scripts/mypythonscript.py

If you have more hosts, repeat on every host.

#!/usr/bin/python3

import json
import sys
import logging

def main():
    # Configure logging, change log file name
    logging.basicConfig(filename='python_event_task_abc.log', level=logging.INFO, format='%(asctime)s - %(message)s')
    
    # Read inputs from stdin
    lines = sys.stdin.read().splitlines()

    if len(lines) != 2:
        print("Error: Expected exactly two lines of input.")
        raise Exception("Expected exactly two lines of input.")

    # Parse the first line, this contains connector variables
    try:
        connector_parameters = json.loads(lines[0])
        host = connector_parameters.get('host', None)
        password = connector_parameters.get('password', None)
        logging.info(f"Connector parameters, Host={host}")
        # read custom connector parameters here
    except json.JSONDecodeError:
        raise Exception("Connector parameters is not a valid JSON string.") 

    # Parse the second line, this contains mappings data from template datacard
    try:
        task_mappings = json.loads(lines[1])
        logging.info(f"Task mappings JSON: {task_mappings}")
        #read task mappings data here

        # Implement your logic here

        
    except json.JSONDecodeError as e:
        raise Exception("Task mappings is not a valid JSON string.") from e
        # Exceptions raised are controlling workflow orchestration node flow, on exception it goes to Exception path, otherwise it goes to Completed path. Exception raised can be seen on Provisioning exception attribute value.
    except Exception as e:
        raise Exception("An unexpected error occurred") from e

if __name__ == "__main__":
    main()

Take customly named script into use to connector

How to take your customly named Python script into use, by selecting it from Connectors UI. Select your script from connectors Provisioning script -dropdown: